II. A summary of standard approaches

We begin with a brief review of some aggregation methods that appear in the earlier literature so that we can later compare their informational requirements with the milder ones that are needed for the application of the Borda rule. Suppose there is a finite set N = {1, . . . , n} of experts (or jurors) with n ≥ 2 members whose task it is to assess the members of a finite set X of m ≥ 2 wines with respect to their relative quality. Typically, the demands on the experts regarding the quality assessment that they are expected to provide are rather strong. In most cases, these jurors are asked to assign a numerical value (a rating) within a predefined range (such as between 0 and 20, for example, as in the case of the so-called Judgment of Paris) to each wine in X. In discussing these methods, we use the notation r_i(x) to indicate the numerical rating of wine x ∈ X by expert i ∈ N.

The first method employs the arithmetic mean of all experts to determine an overall quality rating; this is sometimes referred to as the usual consensus rating. More precisely, for each wine x ∈ X, the arithmetic-mean rating r^AM(x) is defined as

$$r^{AM}( x ) = \displaystyle{1 \over n}\mathop \sum \limits_{i = 1}^n r_i( x ) .$$

A modification of this method can be obtained if a threshold is employed. Formally, let π denote a threshold within the range of possible rating values. The threshold-dependent quality rating $r_i^{TD}$(x) of expert i ∈ N for a wine x ∈ X is

$$r_i^{TD} ( x ) {\rm \;} = \left\{{\matrix{ {r_i( x ) , \;\;if\;r_i( x ) \ge \pi } \cr \;\;\;{0, \;if\;r_i( x ) < \pi } \cr } } \right.$$

and, in analogy with the arithmetic-mean rating, the threshold-dependent arithmetic-mean rating (also known as the proportional approval consensus rating) for a wine x ∈ X is given by

$$r^{TD}( x ) = \displaystyle{1 \over n}\mathop \sum \limits_{i = 1}^n r_i^{TD} \;( x ) .$$

Strictly speaking, these threshold-dependent individual and overall ratings depend on the choice of the threshold π, but we suppress this dependence for ease of exposition; we do not think that this move involves any danger of ambiguity.

An alternative method that eliminates all wines below a threshold π in the individual ratings homogenizes the ratings of the wines that remain so that they all receive the same numerical rating of one. The corresponding approval-vote quality rating $r_i^{AV}$(x) of expert i ∈ N for wine x ∈ X, also referred to as the approval-consensus rating, is defined by

$$r_i^{AV} ( x ) = \left\{{\matrix{ {1, \;\;if\;r_i( x ) \ge \pi } \cr {0, \;if\;r_i( x ) < \pi } \cr } } \right.\;.$$

The overall approval-vote rating r^AV(x) of wine x ∈ X is obtained as the arithmetic mean of these individual ratings so that

$$r^{AV}( x ) = \displaystyle{1 \over n}\mathop \sum \limits_{i = 1}^n r_i^{AV} ( x ) .$$

Again, we suppress the dependence on the threshold π in our notation for the sake of simplicity.

There is one feature that the approval-vote rating method has in common with the Borda method that we advocate—it does not rely on the numerical ratings provided by the experts. However, we think that the approval-vote method goes too far in that regard because it also eliminates all distinctions among the wines that survive the elimination process—all of them are treated equally, even if an expert may have a firm view that some of them are better than others. The Borda method retains this (as we think, vitally important) information regarding the relative quality of the wines under consideration.

The primary motivation behind the use of a threshold is to exclude wines that perform rather poorly in the individual ratings. As we argue later in the paper, the Borda method guarantees that wines with a relatively poor quality rating by all experts do not influence the aggregate ranking of the remaining wines, so that the use of a (somewhat arbitrary and potentially distorting) threshold becomes superfluous.

It is straightforward to define relative variants of the latter two threshold-based methods. The corresponding relative overall ratings are obtained by dividing the original (non-relative) ratings by the sum of the requisite individual threshold-adjusted ratings instead of using the total number of experts in the denominator.

III. Individual wine assessments

The informational requirements on which the methods described in the previous section are based are very demanding; it is by no means obvious that an expert can provide such finely nuanced assessments. In fact, it may very well be the case that even a mere ordering (i.e., a complete and transitive relation) of the wines in X is difficult to elicit from a juror. The method we propose—the Borda method—takes these concerns into account in that it only relies on a very modest amount of information to be provided by the experts. Any complete relation will do; there is no need to require these relations to possess any coherence properties such as transitivity or acyclicity.

Each expert i ∈ N is assumed to have a complete quality relation R_i defined on the set X of wines that are to be assessed. Thus, the statement xR_iy represents the view that wine x ∈ X is at least as good as wine y ∈ X according to expert i ∈ N . As is commonly done, we use xP_iy to denote that wine x is better than wine y according to juror i, and we write xI_iy to indicate that i considers x and y to be equally good. To be precise, the betterness relation P_i and the equal-goodness relation I_i are derived from the at-least-as-good-as relation R_i by letting xP_iy whenever xR_iy and not yR_ix are true, and xI_iy whenever it is the case that xR_iy and yR_ix. We stress that we do not need to assume that the individual quality relations of the experts are transitive; they do not even have to be acyclical.

The quality relations R ₁, . . . , R_n can be collected in a quality profile R = (R ₁, . . . , R_n). Thus, a quality profile consists of n individual quality relations—one relation for each of the experts.

An example may be instructive at this point. Suppose that there are a set of three experts N = {1, 2, 3} and a set of three wines X = {x, y, z}. A quality profile R = (R ₁, R ₂, R ₃) is composed of the individual quality relations given by

$$\eqalign{& yP_1z, \;zP_1x, \;yP_1x, \;\cr & xP_2y, \;yP_2z, \;zP_2x, \;\cr & zI_3x, \;xI_3y, \;zP_3y.} $$

In this example, expert 1 submits an ordering of the three wines, considering y the best wine, followed by z as the second-best, and wine x as the worst. Expert 2's quality relation, on the other hand, cannot be expressed in terms of an ordering because it is cyclical: for this expert, wine x is better than wine y, wine y is better than wine z, and wine z is better than x, leading to a cycle. We do not mean to suggest that such cyclical quality relations are likely to occur, but we include them in the example to emphasize the point that the Borda method is perfectly capable of accommodating such relations. There is an instance of intransitive equal goodness in the quality relation of expert 3: (s)he considers wines z and x to be equally good, and the same judgment applies to wines x and y. In violation of transitivity, however, wine z is considered better than wine y. The relation assigned to expert 3 appears to be quite plausible. As alluded to in the introduction, a quality relation of this nature can easily result from a threshold of perception that prevents the expert from distinguishing wines z and x as well as wines x and y but allows for the unambiguous judgment that z is better than x.

IV. From quality assessments to wine rankings

We now address the issue of aggregating the experts’ quality relations into a ranking of the wines in X. The principle underlying Borda's method has considerable intuitive appeal. For each of the wines x and for each expert i, we count (a) the number of times wine x is judged to be better than one of the other wines and (b) the number of times wine x is judged to be worse than another wine. The difference between the first of these numbers and the second is the individual Borda score of wine x according to expert i. Now, for each wine x, these scores are added over all experts, and the resulting sum is the overall Borda score of wine x. Finally, the overall wine ranking is established by comparing any two wines according to their respective Borda scores. Thus, the criterion to assess the relative overall quality of the wines is the difference between the number of times a wine beats another one in a pairwise contest and the number of times this wine is beaten by another in such a contest. A fundamental and attractive feature of this method is that there is no need whatsoever to invoke any properties of the experts’ quality relations—only pairwise comparisons matter, and these are well-defined for any binary relation.

In our context, a wine-ranking method assigns an ordering (i.e., a complete and transitive relation) on the set of candidate wines X to each quality profile composed of the experts’ individual quality relations. We emphasize that the objective is to rank all possible wines. Anything short of that would be rather unsatisfactory from the viewpoint of a consumer; a cycle or a large degree of non-comparability are attributes that may render the entire ranking exercise close to meaningless. We use R to label the wine ranking that is associated with the profile R = (R ₁, . . . , R_n). The ranking generated by the Borda method is denoted by R^B.

Consider any expert i ∈ N and suppose that his or her quality ranking on the set of wines X is given by R_i. Furthermore, let x ∈ X be any of the wines to be assessed. In line with the informal description provided, the individual Borda score of wine x according to the quality relation R_i of expert i is given by

$$b( {x; \;R_i} ) = \vert \{ z\in X \vert xP_iz\} \vert -\vert \{ z\in X\vert zP_ix\} \vert , \;$$

where the notation |S| is used to indicate the number of elements in the finite set S.

Note that the above difference is unaffected if the quality relation R_i is used in place of the associated betterness relation P_i. This is the case because whenever we have an instance of equal goodness between wines x and y, it follows by definition that xR_iy and yR_ix cancel each other out when calculating the above-defined difference. This argument applies to any two wines, x and y, no matter whether x and y are distinct or identical. As explained in more detail in the following section, the observation that the Borda method possesses this cancellation property is a major reason why there is no need for any thresholds to eliminate the influence of wines that perform poorly according to the quality relations of all experts.

Either or both of the two sets in the definition of the individual Borda scores may be empty. Furthermore, note that these scores are well-defined even if the individual quality relation does not possess any coherence properties such as transitivity; again, this is an important and desirable feature of the Borda method that is not shared by most alternative ranking procedures.

The overall Borda score b(x; R) of a wine x is obtained by adding the individual scores of all experts so that

(1)$$ \matrix{ {b( {x; \;{\bf R}} ) = \mathop \sum \limits_{i = 1}^n b( {x; \;R_i} ) = \mathop \sum \limits_{i = 1}^n ( \vert \{ z \in X\; \vert \;xP_iz\} \vert {\;-\;} \vert \{ z \in X\;\vert \;zP_ix\} \vert ) } \cr { = \mathop \sum \limits_{i = 1}^n \vert \{ z \in X\;\vert \;xP_iz\} \vert \;-\mathop \sum \limits_{i = 1}^n \;\vert \{ z \in X\;\vert \;zP_ix\} \vert .} \cr } \;$$

Finally, the Borda wine ranking R^B is obtained by declaring a wine x to be at least as good as a wine y if and only if the overall Borda score of x is greater than or equal to the score of y, that is,

$$xR^By\Leftrightarrow b( {x; \;{\bf R}} ) \ge b( {y; \;{\bf R}} ) .$$

The method we propose is by no means new, although the arguments we offer here are novel, as they refer to its extension to additional data sets. But the idea that every single instance of betterness has value and should be counted as one of many opinions, all of which have the same value, is stressed in an essay by Morales (Reference Morales, McLean and Urken1797). Morales was a stout defender of Borda's (Reference Borda, McLean and Urken1781) voting rule, on which the Borda method is based. Not everyone was as supportive of Borda as Morales; for example, voting rules founded on the majority principle were advocated by Condorcet (Reference Condorcet, McLean and Urken1785). Daunou (Reference Daunou, McLean and Urken1803), a strong critic of Borda, proposed an alternative voting rule based on a lexicographic combination of the Condorcet criteria and the plurality rule.Footnote ⁵ Following Morales, we also use the term opinion when referring to a single instance of betterness in an expert's assessment of the wines under consideration.

To illustrate the definition of the Borda method, let us return to the example defined in the previous section. It follows that

$$b( {x; \;R_1} ) = 0-2 = {-}2, \;\;\;\;\;b( {y; \;R_1} ) = 2-0 = 2, \;\;\;\;\;\;\;\;b( {z; \;R_1} ) = 1-1 = 0; \;$$

$$b( {x; \;R_2} ) = 1-1 = 0, \;\;\;\;\;\;\;\;b( {y; \;R_2} ) = 1-1 = 0, \;\;\;\;\;\;\;\;b( {z; \;R_2} ) = 1-1 = 0; \;$$

$$b( {x; \;R_3} ) = 0-0 = 0, \;\;\;\;\;\;\;\;b( {y; \;R_3} ) = 0-1 = {-}1, \;\;\;\;\;b( {z; \;R_3} ) = 1-0 = 1.$$

Adding over all experts, we obtain the overall Borda scores

$$b( {x; \;{\bf R}} ) = {-}2 + 0 + 0 = {-}2, \;b( {y; \;{\bf R}} ) = 2 + 0-1 = 1, \;b( {z; \;{\bf R}} ) = 0 + 0 + 1 = 1$$

and, therefore, the overall Borda ranking is given by yI^Bz, yP^Bx, zP^Bx. Thus, wines y and z are tied for the top, and wine x is at the bottom. This example confirms that the Borda method always generates an overall ordering even if the individual quality relations fail to be acyclical.

A few words of explanation may be in order to clarify why two commonly used alternative definitions of the Borda method are not suitable in our setting, owing to the possibility of equal goodness or violations of transitivity in the individual quality relations.

If the experts’ quality relations are strict and complete (so that one of any two distinct wines must be better than the other), the definition of the Borda method can be simplified. The conjunction of these two properties guarantees that the ranking of any two wines is unchanged if the second term in the difference that defines the overall Borda score is omitted. This is the case because the equality

$$\vert \{ z\in X\vert zP_ix\} \vert + \vert \{ z\in X\vert xP_iz\} \vert = m-1$$

is valid for all wines x if R_i is a strict complete relation defined on the set of m wines in X. If individual quality relations may include instances of equal goodness, however, this is not the case—removing the second part of the difference in Equation (1) leads to a different criterion to rank the wines. In the context of quality relations that are not necessarily strict, this second option leads to highly undesirable consequences. For example, suppose that there are three experts N = {1, 2, 3} and three wines X = {x, y, z}. Furthermore, consider the quality profile R = (R ₁, R ₂, R ₃) given by

$$\eqalign{& xI_1y, \;xP_1z, \;yP_1z, \;\cr & xI_2y, \;xI_2z, \;zP_2y, \;\cr & xI_3y, \;xI_3z, \;zP_3y.} $$

According to the Borda method, the overall Borda score of wine x is given by 1 − 0 = 1 because x beats wine z according to expert 1's quality relation, and the overall score of wine y is 1 − 2 = −1 because y beats z according to expert 1 and is beaten by z according to experts 2 and 3. Thus, x is better than y according to the Borda method. However, if we were to count the number of wins only and disregard the number of losses when calculating the requisite scores, x and y would be ranked as equally good because each of them registers one win. This seems difficult to accept because wine y is beaten by z in the quality relations of experts 2 and 3, whereas x does not suffer any such losses. The use of the criterion expressed in Equation (1) avoids troublesome conclusions of this nature.

A second alternative definition of the Borda rule consists of considering it a special case of a scoring method.Footnote ⁶ Assuming that the individual quality relation of each expert is a strict ordering, each wine is given a position: the top wine is in position 1, the next-to-best is in position 2, and so on until we reach the bottom wine, whose position corresponds to m—the total number of wines under consideration. In this environment that is restricted to strict individual orderings, the Borda method can be thought of as assigning a weight to each position, where these weights are given as follows: Position 1 receives a weight of (m − 1) − 0 = m − 1 because this top position is better than the m − 1 remaining ones and worse than none of the others. The weight of position 2 is (m − 2) − 1 = m − 3 because position 2 is superior to m − 2 of the remaining positions and inferior to one position—the top position. This process can be continued until we reach the bottom position m, which is superior to none of the others and inferior to the remaining m − 1 positions, and its weight is 0 − (m − 1) = −(m − 1). Thus, according to the Borda method, the positional weights w ₁, . . . , w_m assigned to the m positions from top to bottom are given by

$$w_1 = m-1, \;w_2 = m-3, \;. . . , \;w_{m-1} = {-}( {m -3} ) , \;w_m = {-}( {m-1} ) .$$

Adding the scores of all experts yields the overall Borda scores as defined in Equation (1). A generalization of this method consists of the class of scoring methods. These are obtained by assigning arbitrary weights w ₁, . . . , w_m to the positions, where the restriction w ₁ ≥ . . . ≥ w_m with at least one strict inequality is usually imposed to ensure that better positions cannot receive lower weights than worse positions. Because the notion of a position cannot even be defined if the experts’ quality relations fail to be orderings, this alternative method cannot be applied in our informationally austere framework. Thus, except for the Borda rule, general scoring methods are not suitable for the purposes of this paper, and, therefore, we cannot treat Borda's method as a special case of these more general rules. Moreover, we note that the Borda rule is perfectly well-defined even if the individual relations are complete and transitive but not necessarily strict—the definition of the criterion does not have to be amended in any way. For other scoring rules, this is not the case because modifications are needed to make sure that weights can be assigned in instances of equal goodness; this is often accomplished by employing arithmetic means to determine the positional weights involved in a tie.

Condorcet's (Reference Condorcet, McLean and Urken1785) method of majority decision shares the flexibility of the Borda rule.Footnote ⁷ It is also applicable if the individual experts’ relations fail to be orderings. According to the Condorcet rule R^C, wine x is at least as good as wine y if and only if the number of experts who rank x higher than y is greater than or equal to the number of experts who rank y higher than x, that is,

$$xR^Cy\Leftrightarrow \vert \{ i\in N \vert xP_iy\} \vert \ge \vert \{ i\in N \vert yP_ix\} \vert .$$

In analogy to the Borda rule, the betterness relation P_i can be replaced with the goodness relation R_i without changing the comparisons according to this method—again, instances of equal goodness cancel each other out. Because only pairwise comparisons appear in this definition, the individual relations do not have to possess any coherence properties such as transitivity. As is well known, the Condorcet method suffers from the serious drawback that it may generate cyclical overall relations, thus making the rule of little use when it comes to providing guidance to consumers. For this reason, we do not advocate Condorcet's method of majority decision; we introduce it merely for comparative purposes in the applied section.

V. Properties of the Borda method

VI. An application to Bordeaux 2021 wines

We conclude with an application of our method to a real-life scenario—the recent tasting of some 2021 red Bordeaux wines gathered on Bordoverview.Footnote ¹¹ This tasting involves ratings for some Bordeaux 2021 future wines produced by five international experts: the Wine Advocate (WA), Jancis Robinson (JR), Jeff Leve (JL), Jane Anson (JA), and Chris Kissack (CK).Footnote ¹²

Table 1 exhibits a selection of seven wines (the most expensive ones), with the ratings assigned by these experts. Because the Borda method merely requires a complete relation provided by each of the experts as the basic input, the only information to be extracted is ordinal in nature. If, for instance, an expert submitted a range such as 95–97, we took this to mean that the wine in question is considered better than a wine with a rating of 95 and worse than a wine with a rating of 97. In one case, there is a rating of 97 for one wine and a rating of 96–98 for another by the same expert, and we treated these two wines as being considered equally good. Analogously, a wine with a rating of 17.5+ was treated as better than a wine with a rating of 17.5. We note that the scales employed by some reviewers differ from those used by others, but this is not a problem because the Borda method only requires ordinal information.

Table 1.

Ratings from five experts for the seven most expensive 2021 red Bordeaux wines

Table 2 presents the individual Borda scores that are obtained by calculating the number of wins minus the number of losses in pairwise comparisons according to each of the experts. For instance, Lafite-Rothschild is assigned an individual Borda score of 4 by expert Jancis Robinson because, according to Table 1, Lafite-Rothschild beats five other wines and is beaten by only one other wine, so the score is given by 5 − 1 = 4.

Table 2.

Individual Borda scores

Table 3 contains the results. The overall Borda scores, obtained by adding the individual Borda scores, are listed first. The final column presents the ranking of the seven wines according to the Borda method. Haut-Brion comes in first with an overall Borda score of 10, followed by Lafite-Rothschild with a score of 9. Then, there is a tie between Margaux and Cheval Blanc (both with a score of 7). Mouton-Rothschild comes in fifth with a score of −4, Palmer is in the next-to-last position with a score of −5 and, finally, Angélus is at the bottom with a score of −24. As alluded to earlier, the Borda scores can serve as wine ratings if desired. We emphasize, however, that these ratings are nothing more than a numerical representation of the Borda ranking.

Table 3.

Overall Borda scores and Borda ranks

Note that both Lafite-Rothschild and Margaux are tied according to the Condorcet criterion. Each is ranked higher than the other by two experts, whereas one juror considers both wines equally good. The same happens to Haut-Brion and Cheval Blanc. However, both Lafite-Rothschild and Margaux win (in a pairwise contest) against Cheval Blanc whereas they both lose against Haut-Brion (in a pairwise contest). Thus, there are two cycles in the aggregate relation. Lafite-Rothschild beats Cheval Blanc; Cheval Blanc and Haut-Brion are tied; and Haut-Brion beats Lafite-Rothschild. The same observation applies if Lafite-Rothschild is replaced with Margaux. In other words, the Condorcet criterion is unable to provide an ordering for the full sample of seven wines, in contrast to the Borda method.

VII. Concluding remarks

The main purpose of this paper is to make a case in favor of using the Borda method in the assessment of wines by a panel of jurors. While this suggestion is not new, a novel aspect of our proposal is the observation that this method is sufficiently flexible to accommodate inputs from the experts that are complete binary relations—properties such as transitivity or merely acyclicity are not required. In principle, completeness can be dispensed with as well; this is the case because the Borda scores are entirely determined based on better-than relations. We retain the completeness assumption to conform to the view that Borda treats all jurors equally, thus allowing all of them to rank all possible pairs of wines. Because only betterness matters, one may be tempted to make the case that the assessments of experts who submit more betterness relations than others have a higher weight in determining the overall ranking. But if this argument is accepted (and we think it should not be), it immediately applies to the Borda method in the traditional framework in which all individual relations are orderings: again, only betterness matters—all instances of equal goodness cancel each other out. Thus, everyone who accepts the Borda method if individual relations are orderings cannot use the equal-treatment argument to reject the Borda method if individual relations are permitted to exhibit instances of non-comparability—the argument is valid either in both environments or in neither of them.

Another issue is the possible view that numerical ratings may be of use when it comes to the comparison of wines that are up for assessment by an expert panel at different time periods. This view rests on the assumption that these ratings are numerically and intertemporally significant: a specific quality rating means the same for every juror, for every wine, and for every time period in which an assessment is performed. To put it another way, a consumer must be confident that a wine that is given a specific rating in one year is, according to the experts being consulted, of the same quality as a wine that receives the same rating in another year. To us, this seems like a rather heroic assumption. As argued earlier, even the assumption that the experts have quality orderings is too demanding in many circumstances; for instance, the possibility of intransitive indifference generated by thresholds of perception is difficult, if not impossible, to rule out. In view of observations of this nature, the hypothesis that every expert can submit a reliable ordering (let alone assign ratings that are numerically significant) may be too much to ask for.

The illustrative example presented in Section VI does not allow us to demonstrate the remarkable flexibility of the Borda method because it utilizes a data set that is generated from individual wine ratings submitted by the experts; therefore, the underlying relations cannot but be orderings. This appears to be inevitable because non-transitive data is not available at this time. We hope, however, that this paper will contribute toward the objective of conducting assessments that permit the experts to submit not necessarily acyclical (thus, not necessarily quasi-transitive or transitive) relations if they so desire.