Skip to main content Accessibility help
×
Home
Hostname: page-component-55597f9d44-5zjcf Total loading time: 0.33 Render date: 2022-08-17T01:20:24.890Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true } hasContentIssue true

Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?

Published online by Cambridge University Press:  31 July 2012

Robert H. Ashton
Affiliation:
L. Palmer Fox Professor, Fuqua School of Business, Duke University.

Abstract

This paper considers the levels of reliability and consensus of wine quality judgments found in studies of experienced wine judges. Both reliability, which concerns the similarity of repeat judgments of a particular wine by the same judge, and consensus, which concerns the similarity of judgments of a particular wine across judges, are necessary requirements for expertise in wine judging. Reliability and consensus levels found in wine judging are compared to those documented by a large body of research in six other fields: medicine, clinical psychology, business, auditing, personnel management, and meteorology. In all fields, including wine judging, reliability is greater than consensus. Both reliability and consensus are, on average, substantially lower in wine judging than in other fields, although tremendous variability exists across judges in every field. Overall, little support is found for the idea that experienced wine judges should be regarded as experts. (JEL Classification: C91)

Type
Research Article
Copyright
Copyright © American Association of Wine Economists 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Amerine, M.A., and Roessler, E.B. (1983). Wines: Their sensory evaluation. New York: W.H. Freeman.Google Scholar
Arens, A.A., Elder, R.J., and Beasley, M.S. (2005). Auditing and assurance services. Upper Saddle River, NJ: Prentice Hall.Google Scholar
Ashenfelter, O., and Quandt, R. (1999). Analyzing a wine tasting statistically. Chance, 12, 1620.CrossRefGoogle Scholar
Ashton, A.H. (1985). Does consensus imply accuracy in accounting studies of decision making? Accounting Review, 60, 173185.Google Scholar
Ashton, R.H. (1999). Enriching the “expertise paradigm” of accounting research: Conscientiousness, general cognitive ability, and goal orientation. Advances in Accounting Behavioral Research, 2, 314.Google Scholar
Ashton, R.H. (2000). A review and analysis of research on the test-retest reliability of professional judgment. Journal of Behavioral Decision Making, 13, 277294.3.0.CO;2-B>CrossRefGoogle Scholar
Ashton, R.H. (2011). Improving experts’ wine quality judgments: Two heads are better than one. Journal of Wine Economics, 6, 160178.CrossRefGoogle Scholar
Ashton, R.H., and Ashton, A.H. (1995). Perspectives on judgment and decision-making research in accounting and auditing. In Ashton, R.H. and Ashton, A.H. (Eds.), Judgment and decision-making research in accounting and auditing. New York: Cambridge University Press. Pages 35.CrossRefGoogle Scholar
Baker, G.A., and Amerine, M.A. (1953). Organoleptic ratings of wines estimated from analytical data. Food Research, 18, 381389.CrossRefGoogle Scholar
Bartoshuk, L.M. (1993). The biological basis of food perception and acceptance. Food Quality and Preference, 4, 2132.CrossRefGoogle Scholar
Bédard, J., and Chi, M.T.H. (1993). Expertise in auditing. Auditing: A Journal of Practice & Theory, 12(Supplement), 2145.Google Scholar
Bouwman, M.J., and Bradley, W.E. (1997). Judgment and decision making, part II: Expertise, consensus and accuracy. In Arnold, V. and Sutton, S.G. (Eds.), Behavioral accounting research: Foundations and frontiers. Sarasota, FL: American Accounting Association. Pages 89133.Google Scholar
Brien, C.J., May, P., and Mayo, O. (1987). Analysis of judge performance in wine-quality evaluations. Journal of Food Science, 52, 12731279.CrossRefGoogle Scholar
Broomell, S.B., and Budescu, D.V. (2009). Why are experts correlated? Decomposing correlations between judges. Psychometrika, 74, 531553.CrossRefGoogle Scholar
Cicchetti, D.V. (2004a). Who won the 1976 blind tasting of French Bordeaux and U.S. cabernets? Parametrics to the rescue. Journal of Wine Research, 15, 211220.CrossRefGoogle Scholar
Cicchetti, D.V. (2004b). On designing experiments and analysing data to assess the reliability and accuracy of blind wine tastings. Journal of Wine Research, 15, 221226.CrossRefGoogle Scholar
Cicchetti, D.V. (2006a). The Paris 1976 wine tastings revisited once more: Comparing ratings of consistent and inconsistent tasters. Journal of Wine Economics, 1, 125140.CrossRefGoogle Scholar
Cicchetti, D.V. (2006b). The 1976 blind wine tastings: On the consistency of tasters from chardonnays to cabernets. Vineyard Data Quantification Society (www.vdqs.net).Google Scholar
Cliff, M.A., and King, M.C. (1997). The evaluation of judges at wine competitions: The application of eggshell plots. Journal of Wine Research, 8, 7580.CrossRefGoogle Scholar
Cooksey, R.W. (1996). Judgment analysis. San Diego: Academic Press.Google Scholar
Davis, E.B., Kennedy, S.J., and Maines, L.A. (2000). The relation between consensus and accuracy in low-to-moderate accuracy tasks: An auditing example. Auditing: A Journal of Practice & Theory, 19, 101121.CrossRefGoogle Scholar
Detre, K.M., Wright, E., Murphy, M.L., and Takaro, T. (1975). Observer agreement in evaluating coronary angiograms. Circulation, 52, 979986.CrossRefGoogle ScholarPubMed
Einhorn, H.J. (1974). Expert judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562571.CrossRefGoogle Scholar
Einhorn, H.J., and Hogarth, R.M. (1981). Rationality and the sanctity of competence. Behavioral and Brain Sciences, 4, 334335.CrossRefGoogle Scholar
Fischhoff, B. (1982). Debiasing. In Kahneman, D., Slovic, P., and Tversky, A. (Eds.), Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press. Pages 422444.CrossRefGoogle Scholar
Gawel, R., and Godden, P.W. (2008). Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research, 14, 18.CrossRefGoogle Scholar
Gawel, R., Royal, A., and Leske, P. (2002). The effect of different oak types on the sensory properties of Chardonnay. Australian and New Zealand Wine Industry Journal, 17, 1420.Google Scholar
Ghiselli, E.E. (1964). Theory of psychological measurement. New York: McGraw-Hill.Google Scholar
Goldberg, L.R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. Psychological Bulletin, 73, 422432.CrossRefGoogle Scholar
Goldwyn, C., and Lawless, H. (1991). How to taste wine (for fun and profit). ASTM Standardization News, 19, 3237.Google Scholar
Goode, J. (2008). Experiencing wine: Why critics mess up (some of the time). In Allhoff, F. (Ed.), Wine & philosophy: A symposium on thinking and drinking. Malden, MA: Blackwell. Pages 137153.Google Scholar
Hodgson, R.T. (2008). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3, 105113.CrossRefGoogle Scholar
Hodgson, R.T. (2009a). An analysis of the concordance among 13 U.S. wine competitions. Journal of Wine Economics, 4, 19.CrossRefGoogle Scholar
Hodgson, R.T. (2009b). How expert are “expert” wine judges? Journal of Wine Economics, 4, 233241.CrossRefGoogle Scholar
Hulkower, N. (2009). The judgment of Paris according to Borda. Journal of Wine Research, 20, 171182.CrossRefGoogle Scholar
Karelaia, N., and Hogarth, R.M. (2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological Bulletin, 134, 404426.CrossRefGoogle ScholarPubMed
Kaufmann, E., and Athanasou, J.A. (2009). A meta-analysis of judgment achievement as defined by the lens model equation. Swiss Journal of Psychology, 68, 99112.CrossRefGoogle Scholar
Keasey, K., and Watson, R. (1989). Consensus and accuracy in accounting studies of decision making: A note on a new measure of consensus. Accounting, Organizations and Society, 14, 337345.CrossRefGoogle Scholar
Kenny, D.A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155163.CrossRefGoogle ScholarPubMed
Lawless, H., Liu, Y., and Goldwyn, C. (1997). Evaluation of wine quality using a small-panel hedonic scaling method. Journal of Sensory Studies, 12, 317332.CrossRefGoogle Scholar
Lee, J.W., and Yates, J.F. (1992). How quantity judgment changes as the number of cues increases: An analytical framework and review. Psychological Bulletin, 12, 363377.CrossRefGoogle Scholar
Lindley, D.V. (2006). Analysis of a wine tasting. Journal of Wine Economics, 1, 3341.CrossRefGoogle Scholar
Lord, F., and Novick, M. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Pincus, K.V. (1990). Audit judgment consensus: A model for dichotomous decisions. Auditing: A Journal of Practice & Theory, 9, 120.Google Scholar
Quandt, R.E. (2006). Measurement and inference in wine tasting. Journal of Wine Economics, 1, 730.CrossRefGoogle Scholar
Quandt, R.E. (2007). A note on a test for the sum of ranksums. Journal of Wine Economics, 2, 98102.CrossRefGoogle Scholar
Schmidt, F.L., and Hunter, J.E. (1992). Development of a causal model of processes determining job performance. Current Directions in Psychological Science, 1, 8992.CrossRefGoogle Scholar
Shanteau, J. (2001). What does it mean when experts disagree? In Salas, E. and Klein, G. (Eds.), Linking expertise and naturalistic decision making. Hillsdale, NJ: Erlbaum. Pages 229244.Google Scholar
Taber, G.M. (2006). Judgment of Paris: California vs. France and the historic 1976 Paris tasting that revolutionized wine. New York: Scribner.Google Scholar
Weiss, D.J., and Shanteau, J. (2004). The vice of consensus and the virtue of consistency. In Smith, K., Shanteau, J., and Johnson, P. (Eds.), Psychological investigations of competence in decision making. Cambridge: Cambridge University Press. Pages 226240.Google Scholar
Wright, W.F. (1988). Audit judgment consensus and experience. In Ferris, K.R. (Ed.), Behavioral accounting research: A critical analysis. Columbus, OH: Century VII. Pages 305328.Google Scholar
37
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *