Hostname: page-component-76d6cb85b7-lrvh5 Total loading time: 0 Render date: 2026-07-15T13:59:45.642Z Has data issue: false hasContentIssue false
  • English
  • Français

Quantifying Randomness Versus Consensus in Wine Quality Ratings*

Published online by Cambridge University Press:  29 April 2014

Jing Cao
Jing Cao*
Affiliation:
Associate Professor, Department of Statistical Science, Southern Methodist University, 6425 Boaz Street, Dallas, TX 75275; e-mail: jcao@mail.smu.edu.
Get access Rights & Permissions [Opens in a new window]

Abstract

There has been ongoing interest in studying wine judges' performance in evaluating wines. Most of the studies have reached a similar conclusion: a significant lack of consensus exists in wine quality ratings. However, a few studies, to the author's knowledge, have provided direct quantification of how much consensus (as opposed to randomness) exists in wine ratings. In this paper, a permutation-based mixed model is proposed to quantify randomness versus consensus in wine ratings. Specifically, wine ratings under the condition of randomness are generated with a permutation method, and wine ratings under the condition of consensus can be produced by sorting the ratings for each judge. Then the observed wine ratings are modeled as a mixture of ratings under randomness and ratings under consensus. This study shows that the model can provide excellent model fit, which indicates that wine ratings, indeed, consist of a mixture of randomness and consensus. A direct measure is easily computed to quantify randomness versus consensus in wine ratings. The method is demonstrated with data analysis from a major wine competition and a simulation study. (JEL Classifications: C10, C13, C15)

Keywords

Mixed modelpermutationquantifying randomnesswine judge consensus

Information

Type
Articles
Information
Journal of Wine Economics , Volume 9 , Issue 2 , August 2014 , pp. 202 - 213
Copyright
Copyright © American Association of Wine Economists 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

*

The author acknowledges support from the California State Fair Commercial Wine Competition for making the data available. Special thanks go to Robert T. Hodgson, G.M. “Pooch” Pucilowski, and Aaron E. Kidder. The author also thanks the reviewer for helpful suggestions that improved the paper.

References

Ashton, J. (2012). Reliability and consensus of experienced wine judges: Expertise within and between? Journal of Wine Economics, 7(1), 70–87.CrossRefGoogle Scholar
Cao, J., and Stokes, L. (2010). Evaluation of wine judge performance through three characteristics: Bias, discrimination, and variation. Journal of Wine Economics, 5(1), 132–142.CrossRefGoogle Scholar
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.Google Scholar
Cui, X., Hwang, J.T.G., Qiu, J., Blades, N.J., and Churchill, G.A. (2005). Improved statistical tests for differential gene expression by shrinking variance components estimates. Biostatistics, 6(1), 59–75.CrossRefGoogle ScholarPubMed
Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.Google Scholar
Gawel, R., and Godden, P.W. (2008). Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research, 14(1), 1–9.CrossRefGoogle Scholar
Hodgson, R.T. (2008). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3(2), 105–113.Google Scholar
Hodgson, R.T. (2009). An analysis of the concordance among 13 U.S. wine competitions. Journal of Wine Economics, 4(1), 1–9.CrossRefGoogle Scholar
McLachlan, G.J., and Peel, D. (2000). Finite Mixture Models. New York: Wiley.CrossRefGoogle Scholar
Pearson, K. (1894). Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society A, 185, 71–110.Google Scholar
Tusher, V.G., Tibshirani, R., and Chu, G. (2001). Significance analysis of microarrays applied to transcriptional responses to ionizing radiation. Proceedings of the National Academy of Sciences, 98(9), 5116–5121.Google Scholar