2. Observed Weighted Agreement

Before correcting for chance agreement, we define observed weighted agreement as

(1) $\begin{array}{c}{A}_w=\frac{1}{N}\sum _{i=1}^N\left({\sum }_{c=1}^C,\frac{R_{i,c}({\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{R}_{i,\tilde{c}}-1)}{R(R-1)}\right)=\frac{{\sum }_{i=1}^N{\sum }_{c=1}^C{R}_{i,c}(\sum _{\tilde{c}=1}^C{w}_{c,\tilde{c}}{R}_{i,\tilde{c}}-1)}{NR(R-1)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{w}=\frac{1}{N}\sum \limits _{i=1}^N \left[ \sum \nolimits _{c=1}^C \frac{R_{i,c}(\sum \nolimits _{\tilde{c}=1}^C w_{c,\tilde{c}} R_{i,\tilde{c}}-1)}{R(R-1)} \right] =\frac{\sum \nolimits _{i=1}^N \sum \nolimits _{c=1}^C {R_{i,c}(\sum \limits _{\tilde{c}=1}^C w_{c,\tilde{c}} R_{i,\tilde{c}}-1)} }{NR(R-1)},\, \end{aligned}$$\end{document}

where N is the number of items, $C\ge 3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C\ge 3$$\end{document} is the number of categories, $R\ge 2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R\ge 2$$\end{document} is the number of raters, and $R_{i,c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{i,c}$$\end{document} is the number of raters who assign item i to category c, with ${\sum }_{c=1}^C{R}_{i,c}=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{c=1}^C R_{i,c} =R$$\end{document} . Furthermore, $w_{c,\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}$$\end{document} defines the weights for pairwise rater (dis)agreements, where c is the category chosen by the first rater, and $\tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}$$\end{document} is chosen by the second rater: $w_{c,\tilde{c}}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}=1$$\end{document} if $c=\tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=\tilde{c}$$\end{document} , and $0\le w_{c,\tilde{c}}<1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0\le w_{c,\tilde{c}}<1$$\end{document} if $c\ne \tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\ne \tilde{c}$$\end{document} (i.e., full credit if the two raters agree, and partial or no credit if they disagree). Because of symmetric weights, $w_{c,\tilde{c}}=w_{\tilde{c},c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}=w_{\tilde{c},c}$$\end{document} for all $c\ne \tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\ne \tilde{c}$$\end{document} . Expression (1) is consistent with Gwet (Reference Gwet2014) and Van Oest and Girard (Reference Van Oest and Girard2021); it reduces to Fleiss (Reference Fleiss1971) if $w_{c,\tilde{c}}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}=0$$\end{document} for all $c\ne \tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\ne \tilde{c}$$\end{document} .

As an example, suppose $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories, $R=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=2$$\end{document} raters, and $N=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=1$$\end{document} item for which the first rater chooses category 1 and the second rater chooses category 2. This situation implies that $R_{1,1}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{1,1}=1$$\end{document} , $R_{1,2}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{1,2}=1$$\end{document} , and $R_{1,3}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{1,3}=0$$\end{document} , so $A_w={\sum }_{c=1}^C{R}_{1,c}({\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{R}_{1,\tilde{c}}-1)/\left[R(R-1)\right]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{w}=\sum \nolimits _{c=1}^C {R_{1,c}(\sum \nolimits _{\tilde{c}=1}^C w_{c,\tilde{c}} R_{1,\tilde{c}}-1)} /[R(R-1)]$$\end{document} becomes $\{1\times (w_{1,1}\times 1+w_{1,2}\times 1+w_{1,3}\times 0-1)+1\times (w_{2,1}\times 1+w_{2,2}\times 1+w_{2,3}\times 0-1)+0\times (w_{3,1}\times 1+w_{3,2}\times 1+w_{3,3}\times 0-1)\}/\{2\times (2-1)\}=\{(w_{1,1}+w_{1,2}-1)+(w_{2,1}+w_{2,2}-1)\}/2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{1\times (w_{1,1}\times 1+w_{1,2}\times 1+w_{1,3}\times 0-1)+1\times (w_{2,1}\times 1+w_{2,2}\times 1+w_{2,3}\times 0-1)+0\times (w_{3,1}\times 1+w_{3,2}\times 1+w_{3,3}\times 0-1)\}/\{2\times (2-1)\}=\{(w_{1,1}+w_{1,2}-1)+(w_{2,1}+w_{2,2}-1)\}/2$$\end{document} . Because $w_{1,1}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{1,1}=1$$\end{document} , $w_{2,2}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{2,2}=1$$\end{document} , and $w_{1,2}=w_{2,1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{1,2}=w_{2,1}$$\end{document} due to symmetry, we obtain $A_w=\{(1+w_{1,2}-1)+(w_{1,2}+1-1)\}/2=w_{1,2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{w}=\{(1+w_{1,2}-1)+(w_{1,2}+1-1)\}/2=w_{1,2}$$\end{document} , which is the weight corresponding to the two categories chosen for the item by the rater pair. More generally, $A_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{w}$$\end{document} in (1) is the average value of $w_{c,\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}$$\end{document} across all combinations of items and rater pairs, which reduces to the proportion of pairwise rater agreement if $w_{c,\tilde{c}}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}=0$$\end{document} for all $c\ne \tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\ne \tilde{c}$$\end{document} .

We consider weighting schemes that penalize rater disagreements based on the distance between the chosen categories, with power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} (Vanbelle Reference Vanbelle2016; Warrens Reference Warrens2013, Reference Warrens2014):

(2) $\begin{array}{c}{w}_{c,\tilde{c}}=1-{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma },\gamma >0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_{c,\tilde{c}}=1-\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma },\, \gamma >0.\, \end{aligned}$$\end{document}

These weights become linear if $\gamma =1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =1$$\end{document} , become quadratic if $\gamma =2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =2$$\end{document} , become radical if $\gamma =.5$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma =.5$$\end{document} , and converge to identity weights (i.e., $w_{c,\tilde{c}}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{c,\tilde{c}}=0$$\end{document} for all $c\ne \tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c\ne \tilde{c}$$\end{document} , implying unweighted agreement) if $\gamma \to 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma \rightarrow 0$$\end{document} . The weighting scheme does not award credit to rater disagreements as $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} approaches zero, whereas it becomes increasingly lenient as $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} increases.

A smaller power parameter (e.g., radical weights) is appropriate for situations where even minor rater disagreements are serious. For example, different teachers may need to do grading and independent regrading of student exams, where one-step deviations are part of the game, but larger deviations quickly become unacceptable. Conversely, a larger power parameter (e.g., quadratic weights) is appropriate if only major rater disagreements are problematic. Furthermore, linear weights are suitable if no obvious arguments exist to deviate from penalization in proportion to the distance of disagreement. For the sake of illustration, we write out the weights $W=\left(w_{c,\tilde{c}}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W=(w_{c,\tilde{c}})$$\end{document} , defined by (2), for $C=5$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=5$$\end{document} categories and different values of the power parameter:

$\begin{array}{cc}& W_{\gamma \to 0\left(\mathrm{identity}\right)}=\left(\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right),\\ & W_{\gamma =.5\left(\mathrm{radical}\right)}=\left(\begin{array}{ccccc}1& .50& .29& .13& 0\\ .50& 1& .50& .29& .13\\ .29& .50& 1& .50& .29\\ .13& .29& .50& 1& .50\\ 0& .13& .29& .50& 1\end{array}\right),\\ & W_{\gamma =1\left(\mathrm{linear}\right)}=\left(\begin{array}{ccccc}1& .75& .50& .25& 0\\ .75& 1& .75& .50& .25\\ .50& .75& 1& .75& .50\\ .25& .50& .75& 1& .75\\ 0& .25& .50& .75& 1\end{array}\right),\\ & W_{\gamma =2\left(\mathrm{quadratic}\right)}=\left(\begin{array}{ccccc}1& .94& .75& .44& 0\\ .94& 1& .94& .75& .44\\ .75& .94& 1& .94& .75\\ .44& .75& .94& 1& .94\\ 0& .44& .75& .94& 1\end{array}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&W_{\gamma \rightarrow 0\mathrm {(identity)}}=\left( {\begin{array}{*{20}c} 1 &{} 0 &{} 0 &{} 0 &{} 0\\ 0 &{} 1 &{} 0 &{} 0 &{} 0\\ 0 &{} 0 &{} 1 &{} 0 &{} 0\\ 0 &{} 0 &{} 0 &{} 1 &{} 0\\ 0 &{} 0 &{} 0 &{} 0 &{} 1\\ \end{array} } \right) , \\&W_{\gamma =.5\mathrm {(radical)}}=\left( {\begin{array}{*{20}c} 1 &{} .50 &{} .29 &{} .13 &{} 0\\ .50 &{} 1 &{} .50 &{} .29 &{} .13\\ .29 &{} .50 &{} 1 &{} .50 &{} .29\\ .13 &{} .29 &{} .50 &{} 1 &{} .50\\ 0 &{} .13 &{} .29 &{} .50 &{} 1\\ \end{array} } \right) , \\&W_{\gamma =1\mathrm {(linear)}}=\left( {\begin{array}{*{20}c} 1 &{} .75 &{} .50 &{} .25 &{} 0\\ .75 &{} 1 &{} .75 &{} .50 &{} .25\\ .50 &{} .75 &{} 1 &{} .75 &{} .50\\ .25 &{} .50 &{} .75 &{} 1 &{} .75\\ 0 &{} .25 &{} .50 &{} .75 &{} 1\\ \end{array} } \right) , \\&W_{\gamma =2\mathrm {(quadratic)}}=\left( {\begin{array}{*{20}c} 1 &{} .94 &{} .75 &{} .44 &{} 0\\ .94 &{} 1 &{} .94 &{} .75 &{} .44\\ .75 &{} .94 &{} 1 &{} .94 &{} .75\\ .44 &{} .75 &{} .94 &{} 1 &{} .94\\ 0 &{} .44 &{} .75 &{} .94 &{} 1\\ \end{array} } \right) . \end{aligned}$$\end{document}

By substituting the weights (2) into observed weighted agreement (1) and subsequently working out the brackets and using that ${\left(\left(c,-,\tilde{c}\right),/,(C-1)\right)}^{\gamma }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( \left| c-\tilde{c} \right| /(C-1) \right) ^{\gamma }=0$$\end{document} if $c=\tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=\tilde{c}$$\end{document} , we obtain

(3) $\begin{array}{cc}{A}_w=& \frac{{\sum }_{i=1}^N{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C\left(1,-,{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma }\right)R_{i,c}{R}_{i,\tilde{c}}-{\sum }_{i=1}^N{\sum }_{c=1}^C{R}_{i,c}}{NR(R-1)}\\ =& \frac{N{R}^2-{\sum }_{c=1}^C{\sum }_{\tilde{c}\ne c}{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma }\left({\sum }_{i=1}^N,R_{i,c}{R}_{i,\tilde{c}}\right)-NR}{NR(R-1)}\\ =& 1-\frac{{\sum }_{c=1}^C{\sum }_{\tilde{c}\ne c}{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma }\left({\sum }_{i=1}^N,R_{i,c}{R}_{i,\tilde{c}}\right)}{NR(R-1)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{w}= & {} \frac{\sum \nolimits _{i=1}^N \sum \nolimits _{c=1}^C {\sum \nolimits _{\tilde{c}=1}^C \left[ 1-\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma } \right] R_{i,c}R_{i,\tilde{c}}} -\sum \nolimits _{i=1}^N \sum \nolimits _{c=1}^C R_{i,c} }{NR(R-1)}\, \nonumber \\= & {} \frac{NR^{2}-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}\ne c} {\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma }\left[ \sum \nolimits _{i=1}^N {R_{i,c}R_{i,\tilde{c}}} \right] } -NR}{NR(R-1)}\, \nonumber \\= & {} 1-\frac{\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}\ne c} {\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma }\left[ \sum \nolimits _{i=1}^N {R_{i,c}R_{i,\tilde{c}}} \right] } }{NR(R-1)}.\, \end{aligned}$$\end{document}

Symmetry in the numerator of (3) regarding c and $\tilde{c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}$$\end{document} (and thus $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} and $\tilde{c}>c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}>c$$\end{document} in $\tilde{c}\ne c)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}\ne c)$$\end{document} implies

(4) $\begin{array}{c}{A}_w=1-\sum _{c=2}^C\sum _{\tilde{c}<c}{\left(\frac{c-\tilde{c}}{C-1}\right)}^{\gamma }{O}_{c,\tilde{c}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{w}=1-\sum \limits _{c=2}^C \sum \limits _{\tilde{c}<c} {\left( \frac{c-\tilde{c}}{C-1} \right) ^{\gamma }O_{c,\tilde{c}}} ,\, \end{aligned}$$\end{document}

where

(5) $\begin{array}{c}{O}_{c,\tilde{c}}=\frac{2\sum _{i=1}^N{R}_{i,c}{R}_{i,\tilde{c}}}{NR(R-1)}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} O_{c,\tilde{c}}=\frac{2\sum \limits _{i=1}^N {R_{i,c}R_{i,\tilde{c}}} }{NR(R-1)}\, \end{aligned}$$\end{document}

is the observed fraction of cases (i.e., combinations of items and rater pairs) for which one rater chooses c and the other rater chooses $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} . In words, (4) states that observed weighted agreement $A_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{w}$$\end{document} equals one minus the total observed weighted disagreement.

3. Chance-Corrected Weighted Agreement

We consider a broad class of chance-corrected weighted agreement coefficients:

(6) $\begin{array}{cc}{I}_w=& \frac{A_w-{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{p}_c{q}_{\tilde{c}}}{1-{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{p}_c{q}_{\tilde{c}}}\\ =& 1-\frac{1-A_w}{1-{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{p}_c{q}_{\tilde{c}}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} I_{w}= & {} \frac{A_{w}-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {w_{c,\tilde{c}}p_{c}q_{\tilde{c}}} }{1-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {w_{c,\tilde{c}}p_{c}q_{\tilde{c}}} }\nonumber \\= & {} 1-\frac{1-A_{w}}{1-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {w_{c,\tilde{c}}p_{c}q_{\tilde{c}}} },\, \end{aligned}$$\end{document}

where the category proportions in the chance correction sum to one and are greater than zero:

(7) $\begin{array}{c}\sum _{c=1}^C{p}_c=\sum _{\tilde{c}=1}^C{q}_{\tilde{c}}=1,p_c>0\mathrm{for}c=1,\dots ,C,q_{\tilde{c}}>0\mathrm{for}\tilde{c}=1,\dots ,C.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{c=1}^C p_{c} =\sum \limits _{\tilde{c}=1}^C q_{\tilde{c}} =1,\, p_{c}>0\,\,\, \mathrm { for\,\, }c=1,\ldots ,C,\, q_{\tilde{c}}>0\,\,\, \mathrm {for\,\, }\tilde{c}=1,\ldots ,C.\, \end{aligned}$$\end{document}

The first part of (7) is logical consistency; the two other parts hold if, for example, all categories are chosen at least once by one of the raters if $p_c=q_c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{c}=q_{c}$$\end{document} for all c or chosen by any two raters (not necessarily for the same item) if $p_c\ne q_c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{c}\ne q_{c}$$\end{document} . The class of coefficients, defined by (6) and (7), is general. For $R=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=2$$\end{document} raters, it includes weighted versions of Cohen’s kappa (Cohen Reference Cohen1960, Reference Cohen1968) and Scott’s pi (Scott Reference Scott1955). For $R\ge 2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R\ge 2$$\end{document} raters, it includes weighted versions of Fleiss’ kappa (Fleiss Reference Fleiss1971), the uniform prior coefficient (Van Oest Reference Van Oest2019; Van Oest and Girard Reference Van Oest and Girard2021), and the S-coefficient (Bennett et al. Reference Bennett, Alpert and Goldstein1954; Brennan and Prediger Reference Brennan and Prediger1981; Warrens Reference Warrens2014). Table 1 provides the operationalizations of $p_c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{c}$$\end{document} and $q_c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q_{c}$$\end{document} for these coefficients.

Although the class of coefficients does not include Krippendorff’s alpha (Gwet, Reference Gwet2014, p.88), this coefficient converges to the weighted Fleiss’ kappa as the number of items N increases. Thus, these coefficients usually provide similar values (Gwet Reference Gwet2014). Another excluded coefficient is the weighted kappa for $R\ge 3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R\ge 3$$\end{document} raters (Mielke et al. Reference Mielke, Berry and Johnston2007, Reference Mielke, Berry and Johnston2009). This coefficient considers the R-dimensional category combinations from all raters together (instead of rater pairs) but is equivalent to a weighted version of Conger’s kappa, where expected weighted agreement ${\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{p}_c{q}_{\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {w_{c,\tilde{c}}p_{c}q_{\tilde{c}}} $$\end{document} becomes the corresponding average across all rater pairs, with rater-specific category proportions $p_c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{c}$$\end{document} and $q_{\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q_{\tilde{c}}$$\end{document} (Conger Reference Conger1980; Warrens Reference Warrens2012b). Furthermore, the class of coefficients excludes Gwet’s AC2 (Gwet, Reference Gwet2014, p.89), which replaces ${\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{w}_{c,\tilde{c}}{p}_c{q}_{\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {w_{c,\tilde{c}}p_{c}q_{\tilde{c}}} $$\end{document} by a substantially different expression.

We substitute the weights (2) and observed weighted agreement (4) into coefficient (6):

(8) $\begin{array}{cc}{I}_w=& 1-\frac{{\sum }_{c=2}^C{\sum }_{\tilde{c}<c}{\left(\frac{c-\tilde{c}}{C-1}\right)}^{\gamma }{O}_{c,\tilde{c}}}{1-{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C\left(1,-,{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma }\right)p_c{q}_{\tilde{c}}}\\ =& 1-\frac{{\sum }_{c=2}^C{\sum }_{\tilde{c}<c}{\left(\frac{c-\tilde{c}}{C-1}\right)}^{\gamma }{O}_{c,\tilde{c}}}{1-{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{p}_c{q}_{\tilde{c}}+{\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{\left(\frac{\left(c,-,\tilde{c}\right)}{C-1}\right)}^{\gamma }{p}_c{q}_{\tilde{c}}}\\ =& 1-\frac{{\sum }_{c=2}^C{\sum }_{\tilde{c}<c}{\left(c,-,\tilde{c}\right)}^{\gamma }{O}_{c,\tilde{c}}}{{\sum }_{c=2}^C{\sum }_{\tilde{c}<c}{\left(c,-,\tilde{c}\right)}^{\gamma }{E}_{c,\tilde{c}}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} I_{w}= & {} 1-\frac{\sum \nolimits _{c=2}^C \sum \nolimits _{\tilde{c}<c} {\left( \frac{c-\tilde{c}}{C-1} \right) ^{\gamma }O_{c,\tilde{c}}} }{1-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {\left[ 1-\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma } \right] p_{c}q_{\tilde{c}}} }\,\nonumber \\= & {} 1-\frac{\sum \nolimits _{c=2}^C \sum \nolimits _{\tilde{c}<c} {\left( \frac{c-\tilde{c}}{C-1} \right) ^{\gamma }O_{c,\tilde{c}}} }{1-\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {p_{c}q_{\tilde{c}}} +\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {\left( \frac{\left| c-\tilde{c} \right| }{C-1} \right) ^{\gamma }p_{c}q_{\tilde{c}}} }\nonumber \\= & {} 1-\frac{\sum \nolimits _{c=2}^C \sum \nolimits _{\tilde{c}<c} {\left( c-\tilde{c} \right) ^{\gamma }O_{c,\tilde{c}}} }{\sum \nolimits _{c=2}^C \sum \nolimits _{\tilde{c}<c} {\left( c-\tilde{c} \right) ^{\gamma }E_{c,\tilde{c}}} },\, \end{aligned}$$\end{document}

where ${\sum }_{c=1}^C{\sum }_{\tilde{c}=1}^C{p}_c{q}_{\tilde{c}}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{c=1}^C \sum \nolimits _{\tilde{c}=1}^C {p_{c}q_{\tilde{c}}} =1$$\end{document} in the middle step of (8) due to property (7), and

(9) $\begin{array}{c}{E}_{c,\tilde{c}}=p_c{q}_{\tilde{c}}+q_c{p}_{\tilde{c}}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{c,\tilde{c}}=p_{c}q_{\tilde{c}}+q_{c}p_{\tilde{c}}\, \end{aligned}$$\end{document}

is the fraction of cases (i.e., combinations of items and rater pairs) expected by chance for which one rater chooses c and the other rater chooses $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} , with $E_{c,\tilde{c}}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{c,\tilde{c}}>0$$\end{document} because of (7). Coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (8) equals one minus the ratio of the total observed and expected weighted disagreements.

To facilitate further interpretation, we recall the definition of $O_{c,\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{c,\tilde{c}}$$\end{document} in (5) and define

(10) $\begin{array}{c}{O}_{\left(l\right)}=\sum _{c=2}^C\sum _{\tilde{c}<c:c-\tilde{c}=l}{O}_{c,\tilde{c}}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} O_{(l)}=\sum \limits _{c=2}^C \sum \limits _{\tilde{c}<c:c-\tilde{c}=l} O_{c,\tilde{c}} \, \end{aligned}$$\end{document}

as the observed fraction of cases for which the categories c and $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} , chosen by two raters, are $l\in \{1,\dots ,C-1\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\in \{1,\ldots ,C-1\}$$\end{document} steps apart; we put brackets around subscript l to emphasize that it refers to the distance between categories. Analogously, recalling the definition of $E_{c,\tilde{c}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{c,\tilde{c}}$$\end{document} in (9), we define

(11) $\begin{array}{c}{E}_{\left(l\right)}=\sum _{c=2}^C\sum _{\tilde{c}<c:c-\tilde{c}=l}{E}_{c,\tilde{c}}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{(l)}=\sum \limits _{c=2}^C \sum \limits _{\tilde{c}<c:c-\tilde{c}=l} E_{c,\tilde{c}} \, \end{aligned}$$\end{document}

as the fraction of cases expected by chance for which the categories c and $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} , obtained from two raters, are l steps apart. Table 1 provides the exact expressions of (11) for the coefficients. We note that $E_{\left(l\right)}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{(l)}>0$$\end{document} because of (7). Using (10) and (11), we rewrite the chance-corrected weighted agreement coefficient (8) in terms of all possible category distances and their observed and expected frequencies:

(12) $\begin{array}{c}{I}_w=1-\frac{{\sum }_{l=1}^{C-1}{l}^{\gamma }{O}_{\left(l\right)}}{{\sum }_{l=1}^{C-1}{l}^{\gamma }{E}_{\left(l\right)}}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} I_{w}=1-\frac{\sum \nolimits _{l=1}^{C-1} {l^{\gamma }O_{\left( l \right) }} }{\sum \nolimits _{l=1}^{C-1} {l^{\gamma }E_{\left( l \right) }} }.\, \end{aligned}$$\end{document}

Table 1. Operationalizations of coefficients within considered class.

C is the number of categories, $C-l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C-l$$\end{document} is the number of combinations $(c,\tilde{c})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c,\tilde{c})$$\end{document} with $\tilde{c}<c$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{c}<c$$\end{document} and $c-\tilde{c}=l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c-\tilde{c}=l$$\end{document} , N is the number of items, R is the number of raters, $R_{i,c}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{i,c}$$\end{document} is the number of raters who assign item i to category c, $R_{i,c}^{\left(1\right)}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{i,c}^{(1)}=1$$\end{document} if rater 1 assigns item i to category c (zero else), and $R_{i,c}^{\left(2\right)}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{i,c}^{(2)}=1$$\end{document} if rater 2 assigns item i to category c (zero else). For later reference: Theorems 1 and 2, and Corollaries 1, 2, 3, 7, and 8 apply to all coefficients, with Corollaries 2 and 8 pertaining to $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories. Furthermore, Corollaries 4, 5, 6, and 9 specifically apply to the S-coefficient, with Corollaries 5 and 9 pertaining to $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories.

4. First-Order Derivative

The first-order derivative of $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} with respect to $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} describes coefficient susceptibility, that is, the direction and degree to which the coefficient value changes as the power parameter of the weighting scheme increases.

Theorem 1

The first-order derivative of coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) with respect to power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) is

(13) $\begin{array}{c}\frac{d{I}_w}{d\gamma }=\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)\left(\frac{l^{\gamma }{E}_{\left(l\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\frac{m^{\gamma }{E}_{\left(m\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\frac{O_{\left(l\right)}}{E_{\left(l\right)}},-,\frac{O_{\left(m\right)}}{E_{\left(m\right)}}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{dI_{w}}{d\gamma }=\sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) \left( \frac{l^{\gamma }E_{\left( l \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) \left( \frac{m^{\gamma }E_{\left( m \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) \left\{ \frac{O_{\left( l \right) }}{E_{\left( l \right) }}-\frac{O_{\left( m \right) }}{E_{\left( m \right) }} \right\} } ,\, \end{aligned}$$\end{document}

where we obtained coefficient (12) from (1), (2), (6), and (7), the summation indices l, m, and s are category distances, $O_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{\left( l \right) }$$\end{document} is defined by (10) and (5), and $E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{\left( l \right) }$$\end{document} is defined by (11) and (9).

Proof

Differentiating (12) with respect to $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} yields

$\begin{array}{cc}\frac{d{I}_w}{d\gamma }=& \frac{d\left(1,-,\frac{{\sum }_{l=1}^{C-1}{l}^{\gamma }{O}_{\left(l\right)}}{{\sum }_{l=1}^{C-1}{l}^{\gamma }{E}_{\left(l\right)}}\right)}{d\gamma }\\ =& \frac{-\left({\sum }_{l=1}^{C-1},l^{\gamma }ln\left(l\right)O_{\left(l\right)}\right)\left({\sum }_{m=1}^{C-1},m^{\gamma }{E}_{\left(m\right)}\right)+\left({\sum }_{l=1}^{C-1},l^{\gamma }{O}_{\left(l\right)}\right)\left({\sum }_{m=1}^{C-1},m^{\gamma }ln\left(m\right)E_{\left(m\right)}\right)}{{\left({\sum }_{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2}\\ =& \frac{-{\sum }_{m=1}^{C-1}{\sum }_{l=1}^{C-1}ln\left(l\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)}+{\sum }_{m=1}^{C-1}{\sum }_{l=1}^{C-1}ln\left(m\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)}}{{\left({\sum }_{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \, \frac{dI_{w}}{d\gamma }= & {} \frac{d\left( 1-\frac{\sum \nolimits _{l=1}^{C-1} {l^{\gamma }O_{\left( l \right) }} }{\sum \nolimits _{l=1}^{C-1} {l^{\gamma }E_{\left( l \right) }} } \right) }{d\gamma } \\= & {} \frac{-\, \left\{ \sum \nolimits _{l=1}^{C-1} {l^{\gamma }\ln \left( l \right) O_{\left( l \right) }} \right\} \left\{ \sum \nolimits _{m=1}^{C-1} {m^{\gamma }E_{\left( m \right) }} \right\} +\left\{ \sum \nolimits _{l=1}^{C-1} {l^{\gamma }O_{\left( l \right) }} \right\} \left\{ \sum \nolimits _{m=1}^{C-1} {m^{\gamma }\ln \left( m \right) E_{\left( m \right) }} \right\} }{\left( \sum \nolimits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}} \\= & {} \frac{-\sum \nolimits _{m=1}^{C-1} \sum \nolimits _{l=1}^{C-1} {\ln \left( l \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} +\sum \nolimits _{m=1}^{C-1} \sum \nolimits _{l=1}^{C-1} {\ln \left( m \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} }{\left( \sum \nolimits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}},\, \end{aligned}$$\end{document}

where the notation uses different indices l, m, and s in the summations to allow for combining. Using that $-ln\left(l\right)+\ln \left(m\right)=\ln (m/l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-\ln \left( l \right) +\mathrm {ln}\left( m \right) =\mathrm {ln}(m/l)$$\end{document} and that $\ln (m/l)=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}(m/l)=0$$\end{document} if $l=m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=m$$\end{document} , we obtain

$\begin{array}{c}\frac{d{I}_w}{d\gamma }=\left(\sum _{m=1}^{C-1},\sum _{l=1,l\ne m}^{C-1},\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)}\right)/{\left(\sum _{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{dI_{w}}{d\gamma }=\left( \sum \limits _{m=1}^{C-1} \sum \limits _{l=1,l\ne m}^{C-1} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} \right) /\left( \sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}.\, \end{aligned}$$\end{document}

Next, we decompose $l\ne m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\ne m$$\end{document} into $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} and $l>m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l>m$$\end{document} :

$\begin{array}{cc}\frac{d{I}_w}{d\gamma }=& \left(\sum _{m=1}^{C-1},\sum _{l<m},\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)},+,\sum _{m=1}^{C-1},\sum _{l>m},\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)}\right)/{\left(\sum _{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2\\ =& \left(\sum _{m=1}^{C-1},\sum _{l<m},\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{O}_{\left(l\right)}{E}_{\left(m\right)},+,\sum _{m=1}^{C-1},\sum _{l<m},\ln \left(\frac{l}{m}\right)m^{\gamma }{l}^{\gamma }{O}_{\left(m\right)}{E}_{\left(l\right)}\right)/{\left(\sum _{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2\\ =& \left(\sum _{m=1}^{C-1},\sum _{l<m},\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }\left(O_{\left(l\right)},E_{\left(m\right)},-,O_{\left(m\right)},E_{\left(l\right)}\right)\right)/{\left(\sum _{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{dI_{w}}{d\gamma }= & {} \left( \sum \limits _{m=1}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} +\sum \limits _{m=1}^{C-1} \sum \limits _{l>m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} \right) /\left( \sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}\\= & {} \left( \sum \limits _{m=1}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }O_{\left( l \right) }E_{\left( m \right) }} +\sum \limits _{m=1}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{l}{m} \right) m^{\gamma }l^{\gamma }O_{\left( m \right) }E_{\left( l \right) }} \right) /\left( \sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2} \\= & {} \left( \sum \limits _{m=1}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }\left\{ O_{\left( l \right) }E_{\left( m \right) }-O_{\left( m \right) }E_{\left( l \right) } \right\} } \right) /\left( \sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}.\, \end{aligned}$$\end{document}

Using that $m=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m=1$$\end{document} is infeasible if $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} , and rewriting yields the result, completing the proof. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

The first-order derivative in (13) is a weighted sum taken over all pairs of different category distances m and $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} . As reflected by the term $(O_{\left(l\right)}/E_{\left(l\right)})-(O_{\left(m\right)}/E_{\left(m\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{\left( l \right) }/E_{\left( l \right) })-\mathrm {\, }(O_{\left( m \right) }/E_{\left( m \right) })$$\end{document} , each component compares its smaller distance l with its larger distance m in terms of the ratio of observed to expected-by-chance frequency. Because all other terms in (13) are strictly positive, it holds that $d{I}_w/d\gamma >0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma >0$$\end{document} (i.e., coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} is increasing in power parameter $\gamma )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma )$$\end{document} if the ratio of observed to expected-by-chance frequency tends to decrease as categories are farther apart; that is, if mostly $O_{\left(l\right)}/E_{\left(l\right)}>O_{\left(m\right)}/E_{\left(m\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{\left( l \right) }/E_{\left( l \right) }>O_{\left( m \right) }/E_{\left( m \right) }$$\end{document} for $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} . However, violations are allowed because of the compensatory structure in the weighted sum.

In (13), a component’s comparison of category distances becomes more important in shaping the first-order derivative as (i) the ratio of the larger versus smaller distance increases (via $\ln \left(m,/,l\right))$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}\left( m/l \right) )$$\end{document} , and (ii) these two distances capture greater shares of the total expected weighted disagreement across all distances (via the fractions $l^{\gamma }{E}_{\left(l\right)}/\left(\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l^{\gamma }E_{\left( l \right) }/(\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} )$$\end{document} and $m^{\gamma }{E}_{\left(m\right)}/\left(\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}\right))$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m^{\gamma }E_{\left( m \right) }/(\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} ))$$\end{document} . Furthermore, the latter implies that a component’s importance increases as its two distances l and m are more likely to occur by chance (via $E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{\left( l \right) }$$\end{document} and $E_{\left(m\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{\left( m \right) })$$\end{document} , and these distances l and m increase, where higher values of $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} play a reinforcing role (via $l^{\gamma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l^{\gamma }$$\end{document} and $m^{\gamma })$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m^{\gamma })$$\end{document} . For example, in a two-rater contingency table, the elements far from the main diagonal increasingly determine how the coefficient value responds to changes in the power parameter as this parameter increases. An implication is that the relationship between the power parameter and the coefficient value can be non-monotonic, as changes in $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} trigger shifts in the importance of components that compare different category distances, with possibly opposite contributions via the signs of $(O_{\left(l\right)}/E_{\left(l\right)})-(O_{\left(m\right)}/E_{\left(m\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{\left( l \right) }/E_{\left( l \right) })-\mathrm {\, }(O_{\left( m \right) }/E_{\left( m \right) })$$\end{document} . Furthermore, the log-ratio of category distances $\ln (m/l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}(m/l)$$\end{document} in (13) implies that the degree of coefficient susceptibility is often higher in settings with more categories (i.e., higher C). The reason is that $\ln (m/l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}(m/l)$$\end{document} tends to take higher values as C increases, magnifying the effects of the comparisons $(O_{\left(l\right)}/E_{\left(l\right)})-(O_{\left(m\right)}/E_{\left(m\right)})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{\left( l \right) }/E_{\left( l \right) })-\mathrm {\, }(O_{\left( m \right) }/E_{\left( m \right) })$$\end{document} .

5. Conditions for Direction of Coefficient Susceptibility

We obtain a necessary and sufficient condition from (13) in Theorem 1:

Corollary 1a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) increases if and only if

$\begin{array}{c}\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{E}_{\left(l\right)}{E}_{\left(m\right)}\left(\frac{O_{\left(l\right)}}{E_{\left(l\right)}},-,\frac{O_{\left(m\right)}}{E_{\left(m\right)}}\right)>0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }E_{\left( l \right) }E_{\left( m \right) }\left\{ \frac{O_{\left( l \right) }}{E_{\left( l \right) }}-\frac{O_{\left( m \right) }}{E_{\left( m \right) }} \right\} } >0. \end{aligned}$$\end{document}

Corollary 1b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) decreases if and only if

$\begin{array}{c}\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }{E}_{\left(l\right)}{E}_{\left(m\right)}\left(\frac{O_{\left(l\right)}}{E_{\left(l\right)}},-,\frac{O_{\left(m\right)}}{E_{\left(m\right)}}\right)<0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }E_{\left( l \right) }E_{\left( m \right) }\left\{ \frac{O_{\left( l \right) }}{E_{\left( l \right) }}-\frac{O_{\left( m \right) }}{E_{\left( m \right) }} \right\} } <0. \end{aligned}$$\end{document}

Proof

We have ${\sum }_{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} >0$$\end{document} due to property (7), so we may ignore these ${\sum }_{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum \nolimits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} $$\end{document} terms that do not determine the sign of $d{I}_w/d\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma $$\end{document} in (13). Thus, $d{I}_w/d\gamma >0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma >0$$\end{document} is equivalent to the simpler condition in Corollary 1a, and $d{I}_w/d\gamma <0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma <0$$\end{document} is equivalent to the condition in Corollary 1b. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

As before, each component compares its two category distances m and $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} in terms of the ratio of observed to expected-by-chance frequency, and components comparing larger distances become relatively more important as the power parameter increases (via $l^{\gamma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l^{\gamma }$$\end{document} and $m^{\gamma })$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m^{\gamma })$$\end{document} .

For $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories, the necessary and sufficient condition in Corollary 1 becomes particularly simple:

Corollary 2a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases in settings with three categories, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) increases if and only if

$\begin{array}{c}{O}_{\left(1\right)}/E_{\left(1\right)}>O_{\left(2\right)}/E_{\left(2\right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} O_{\left( 1 \right) }/E_{\left( 1 \right) }>O_{\left( 2 \right) }/E_{\left( 2 \right) }{.} \end{aligned}$$\end{document}

Corollary 2b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases in settings with three categories, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) decreases if and only if

$\begin{array}{c}{O}_{\left(1\right)}/E_{\left(1\right)}<O_{\left(2\right)}/E_{\left(2\right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} O_{\left( 1 \right) }/E_{\left( 1 \right) }<O_{\left( 2 \right) }/E_{\left( 2 \right) }{.} \end{aligned}$$\end{document}

Proof

The only feasible pair of category distances with $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} for $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories corresponds to $l=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=1$$\end{document} and $m=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m=2$$\end{document} . Because $\ln \left(m,/,l\right)l^{\gamma }{m}^{\gamma }{E}_{\left(l\right)}{E}_{\left(m\right)}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}\left( m/l \right) l^{\gamma }m^{\gamma }E_{\left( l \right) }E_{\left( m \right) }>0$$\end{document} , substituting $l=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=1$$\end{document} and $m=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m=2$$\end{document} into Corollary 1 yields Corollary 2.

Corollary 2 implies that the relationship between $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} and $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} is always monotonic (either increasing or decreasing) for $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories. The direction is determined by whether the ratio of observed to expected-by-chance frequency is greater for combinations of categories that are one step apart or two steps apart.

Furthermore, we obtain a sufficient condition that extends the sufficient condition by Warrens (Reference Warrens2013) beyond two raters:

Corollary 3a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) increases if the ratio $O_{\left(l\right)}/E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/E_{(l)}$$\end{document} is decreasing in the category distance l.

Corollary 3b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the chance-corrected weighted agreement coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) decreases if the ratio $O_{\left(l\right)}/E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/E_{(l)}$$\end{document} is increasing in the category distance l.

Proof

Because $\ln \left(m,/,l\right)l^{\gamma }{m}^{\gamma }{E}_{\left(l\right)}{E}_{\left(m\right)}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}\left( m/l \right) l^{\gamma }m^{\gamma }E_{\left( l \right) }E_{\left( m \right) }>0$$\end{document} , the necessary and sufficient condition for $d{I}_w/d\gamma >0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma >0$$\end{document} in Corollary 1a is satisfied if $(O_{\left(l\right)}/E_{\left(l\right)})-(O_{\left(m\right)}/E_{\left(m\right)})>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{(l)}/E_{(l)})-(O_{(m)}/E_{(m)})>0$$\end{document} for all $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} ; that is, if $O_{\left(l\right)}/E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/E_{(l)}$$\end{document} is decreasing in l. Similarly, the necessary and sufficient condition for $d{I}_w/d\gamma <0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma <0$$\end{document} in Corollary 1b is satisfied if $(O_{\left(l\right)}/E_{\left(l\right)})-(O_{\left(m\right)}/E_{\left(m\right)})<0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{(l)}/E_{(l)})-(O_{(m)}/E_{(m)})<0$$\end{document} for all $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} ; that is, if $O_{\left(l\right)}/E_{\left(l\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/E_{(l)}$$\end{document} is increasing in l. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Thus, the relationship between power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} and coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} is monotonic if the ratio of observed to expected-by-chance frequency is monotonic in the category distance. The sufficient condition in Corollary 3 is both necessary and sufficient in three-category settings due to Corollary 2.

6. Conditions for Direction of Coefficient Susceptibility: Weighted S-coefficient

It is instructive to apply the three corollaries to the weighted S-coefficient, which assumes that all C categories are equally likely to occur by chance; that is, $p_c=q_c=1/C$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{c}=q_{c}=1/C$$\end{document} , $c=1,\dots ,C$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c=1,\ldots ,C$$\end{document} , and hence $E_{\left(l\right)}=2(C-l)/C^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{\left( l \right) }=2(C-l)/C^{2}$$\end{document} (see Table 1). We note that $C-l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C-l$$\end{document} is the number of category combinations $(c,\tilde{c})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c,\tilde{c})$$\end{document} with distance l; that is, satisfying $c-\tilde{c}=l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c-\tilde{c}=l$$\end{document} . By applying Corollary 1 to the weighted S-coefficient, we obtain a necessary and sufficient condition for this coefficient:

Corollary 4a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the weighted S-coefficient increases if and only if

$\begin{array}{c}\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }(C-l)(C-m)\left(,\frac{O_{\left(l\right)}}{C-l},-,\frac{O_{\left(m\right)}}{C-m},\right)>0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }(C-l)(C-m)\left\{ \, \frac{O_{\left( l \right) }}{C-l}-\frac{O_{\left( m \right) }}{C-m}\, \right\} } >0. \end{aligned}$$\end{document}

Corollary 4b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the weighted S-coefficient decreases if and only if

$\begin{array}{c}\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)l^{\gamma }{m}^{\gamma }(C-l)(C-m)\left(,\frac{O_{\left(l\right)}}{C-l},-,\frac{O_{\left(m\right)}}{C-m},\right)<0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) l^{\gamma }m^{\gamma }(C-l)(C-m)\left\{ \, \frac{O_{\left( l \right) }}{C-l}-\frac{O_{\left( m \right) }}{C-m}\, \right\} } <0. \end{aligned}$$\end{document}

Proof

Substituting $E_{\left(l\right)}=2(C-l)/C^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{\left( l \right) }=2(C-l)/C^{2}$$\end{document} into Corollary 1 and ignoring the positive constant term 2/ $C^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^{2}$$\end{document} (that does not affect the sign) yields the result. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

As before, this condition considers a weighted sum taken over all pairs of different category distances m and $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} . As reflected by the term $(O_{\left(l\right)}/(C-l))-(O_{\left(m\right)}/(C-m))$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{\left( l \right) }/(C-l))-(O_{\left( m \right) }/(C-m))$$\end{document} , each component compares its smaller distance l with its larger distance m in terms of the average observed frequency per category combination $(c,\tilde{c})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c,\tilde{c})$$\end{document} . Component importance increases as more category combinations $(c,\tilde{c})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c,\tilde{c})$$\end{document} have the corresponding distances l and m (via $C-l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C-l$$\end{document} and $C-m)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C-m)$$\end{document} , the ratio of the larger versus smaller distance increases (via $\ln \left(m,/,l\right))$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}\left( m/l \right) )$$\end{document} , and these distances themselves increase, where higher values of $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} play a reinforcing role (via $l^{\gamma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l^{\gamma }$$\end{document} and $m^{\gamma })$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m^{\gamma })$$\end{document} .

Analogous to Corollary 2, the necessary and sufficient condition for the weighted S-coefficient in Corollary 4 becomes particularly simple if there are only $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories:

Corollary 5a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases in settings with three categories, the weighted S-coefficient increases if and only if $O_{\left(1\right)}/2>O_{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{\left( 1 \right) }/2>O_{\left( 2 \right) }$$\end{document} .

Corollary 5b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases in settings with three categories, the weighted S-coefficient decreases if and only if $O_{\left(1\right)}/2<O_{\left(2\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{\left( 1 \right) }/2<O_{\left( 2 \right) }$$\end{document} .

Proof

The only feasible pair of category distances with $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} for $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories corresponds to $l=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=1$$\end{document} and $m=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m=2$$\end{document} . Because $\ln \left(m,/,l\right)l^{\gamma }{m}^{\gamma }\left(C,-,l\right)\left(C,-,m\right)>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}\left( m/l \right) l^{\gamma }m^{\gamma }\left( C-l \right) \left( C-m \right) >0$$\end{document} , substituting $l=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l=1$$\end{document} , $m=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m=2$$\end{document} , and $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} into Corollary 4 yields Corollary 5. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Corollary 5 implies that the relationship between the power parameter and the weighted S-

coefficient is always monotonic (either increasing or decreasing) for $C=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C=3$$\end{document} categories. The direction is determined by whether the average observed frequency of the two category combinations $(c=2,\tilde{c}=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c=2,\tilde{c}=1)$$\end{document} and $(c=3,\tilde{c}=2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c=3,\tilde{c}=2)$$\end{document} , with distance one, is greater than the observed frequency of category combination $(c=3,\tilde{c}=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(c=3,\tilde{c}=1)$$\end{document} , with distance two, or not.

Furthermore, we obtain a sufficient condition for the weighted S-coefficient that extends a sufficient condition by Warrens (Reference Warrens2014) beyond two raters:

Corollary 6a

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the weighted S-coefficient increases if the average observed frequency $O_{\left(l\right)}/(C-l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/(C-l)$$\end{document} is decreasing in distance l.

Corollary 6b

As power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) increases, the weighted S-coefficient decreases if the average observed frequency $O_{\left(l\right)}/(C-l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/(C-l)$$\end{document} is increasing in distance l.

Proof

Because $\ln (m/l)l^{\gamma }{m}^{\gamma }(C-l)(C-m)>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ln}(m/l)l^{\gamma }m^{\gamma }(C-l)(C-m)>0$$\end{document} , the necessary and sufficient condition for $d{I}_w/d\gamma >0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma >0$$\end{document} in Corollary 4a is satisfied if $(O_{\left(l\right)}/(C-l))-(O_{\left(m\right)}/(C-m))>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{(l)}/(C-l))-(O_{(m)}/(C-m))>0$$\end{document} for all $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} ; that is, if $O_{\left(l\right)}/(C-l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/(C-l)$$\end{document} is decreasing in l. Similarly, the necessary and sufficient condition for $d{I}_w/d\gamma <0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$dI_{w}/d\gamma <0$$\end{document} in Corollary 4b is satisfied if $(O_{\left(l\right)}/(C-l))-(O_{\left(m\right)}/(C-m))<0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(O_{(l)}/(C-l))-(O_{(m)}/(C-m))<0$$\end{document} for all $l<m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l<m$$\end{document} ; that is, if $O_{\left(l\right)}/(C-l)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{(l)}/(C-l)$$\end{document} is increasing in l. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

Thus, the relationship between the power parameter and the weighted S-coefficient is monotonic if the average observed frequency per category combination is monotonic in the category distance. The sufficient condition in Corollary 6 is both necessary and sufficient in three-category settings due to Corollary 5.

7. Second-Order Derivative

The second-order derivative of $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} with respect to $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} helps describe change in coefficient susceptibility, that is, whether the coefficient’s susceptibility to the power parameter of the weighting scheme intensifies or weakens as this parameter increases.

Theorem 2

The second-order derivative of coefficient $I_w$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{w}$$\end{document} in (12) with respect to power parameter $\gamma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} in weighting scheme (2) is

(14) $\begin{array}{c}\frac{d^2{I}_w}{{\left(d\gamma \right)}^2}=\sum _{m=2}^{C-1}\sum _{l<m}\ln \left(\frac{m}{l}\right)\left(\frac{l^{\gamma }{E}_{\left(l\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\frac{m^{\gamma }{E}_{\left(m\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\frac{\sum _{s=1}^{C-1}\ln (lm/s^2)s^{\gamma }{E}_{\left(s\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\frac{O_{\left(l\right)}}{E_{\left(l\right)}},-,\frac{O_{\left(m\right)}}{E_{\left(m\right)}}\right).\\ \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{d^{2}I_{w}}{{(d\gamma )}^{2}}=\sum \limits _{m=2}^{C-1} \sum \limits _{l<m} {\mathrm {ln}\left( \frac{m}{l} \right) \left( \frac{l^{\gamma }E_{\left( l \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) \left( \frac{m^{\gamma }E_{\left( m \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) \left\{ \frac{\sum \limits _{s=1}^{C-1} {\mathrm {ln(}lm\mathrm {/}s^{2}\mathrm {)}s^{\gamma }E_{\left( s \right) }} }{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right\} \left\{ \frac{O_{\left( l \right) }}{E_{\left( l \right) }}-\frac{O_{\left( m \right) }}{E_{\left( m \right) }} \right\} } .\nonumber \\ \end{aligned}$$\end{document}

Proof

Starting from the first-order derivative in (13), we first note that

$\begin{array}{cc}\frac{d\left(\frac{l^{\gamma }{E}_{\left(l\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)}{d\gamma }=& \frac{l^{\gamma }\ln \left(l\right)E_{\left(l\right)}\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}-l^{\gamma }{E}_{\left(l\right)}\sum _{s=1}^{C-1}{s}^{\gamma }ln\left(s\right)E_{\left(s\right)}}{{\left(\sum _{s=1}^{C-1},s^{\gamma }{E}_{\left(s\right)}\right)}^2}\\ =& \left(\frac{l^{\gamma }{E}_{\left(l\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right)\left(\ln ,\left(l\right),-,\frac{\sum _{s=1}^{C-1}\ln \left(s\right)s^{\gamma }{E}_{\left(s\right)}}{\sum _{s=1}^{C-1}{s}^{\gamma }{E}_{\left(s\right)}}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{d\left( \frac{l^{\gamma }E_{\left( l \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) }{d\gamma }= & {} \frac{l^{\gamma }\mathrm {ln} \left( l \right) E_{\left( l \right) }\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} -l^{\gamma }E_{\left( l \right) }\sum \limits _{s=1}^{C-1} {s^{\gamma }\ln \left( s \right) E_{\left( s \right) }} }{\left( \sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} \right) ^{2}} \\= & {} \left( \frac{l^{\gamma }E_{\left( l \right) }}{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right) \left\{ \mathrm {ln}\left( l \right) -\frac{\sum \limits _{s=1}^{C-1} {\mathrm {ln}(s)s^{\gamma }E_{\left( s \right) }} }{\sum \limits _{s=1}^{C-1} {s^{\gamma }E_{\left( s \right) }} } \right\} , \end{aligned}$$\end{document}

so that