Hostname: page-component-76fb5796d-vvkck Total loading time: 0 Render date: 2024-04-27T17:46:02.095Z Has data issue: false hasContentIssue false
Accepted manuscript

Dead rats, dopamine, performance metrics, and peacock tails: proxy failure is an inherent risk in goal-oriented systems

Published online by Cambridge University Press:  26 June 2023

Yohan J. John
Affiliation:
Neural Systems Laboratory, Department of Health and Rehabilitation Sciences, Boston University, Boston, USA, yohan@bu.edu
Leigh Caldwell
Affiliation:
Irrational Agency, London, UK, leigh@irrationalagency.com
Dakota E. McCoy
Affiliation:
Stanford University, Stanford, USA, mccoy6@stanford.edu
Oliver Braganza
Affiliation:
Institute for Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany Institute for Socioeconomics, University of Duisburg-Essen, Duisburg, Germany, oliver.braganza@ukbonn.de

Abstract

When a measure becomes a target, it ceases to be a good measure. For example, when standardized test scores in education become targets, teachers may start ‘teaching to the test’, leading to breakdown of the relationship between the measure--test performance--and the underlying goal--quality education. Similar phenomena have been named and described across a broad range of contexts, such as economics, academia, machine-learning, and ecology. Yet it remains unclear whether these phenomena bear only superficial similarities, or if they derive from some fundamental unifying mechanism. Here, we propose such a unifying mechanism, which we label proxy failure. We first review illustrative examples and their labels, such as the ‘Cobra effect’, ‘Goodhart's law’, and ‘Campbell's law’. Second, we identify central prerequisites and constraints of proxy failure, noting that it is often only a partial failure or divergence. We argue that whenever incentivization or selection is based on an imperfect proxy measure of the underlying goal, a pressure arises which tends to make the proxy a worse approximation of the goal. Third, we develop this perspective for three concrete contexts, namely neuroscience, economics and ecology, highlighting similarities and differences. Fourth, we outline consequences of proxy failure, suggesting it is key to understanding the structure and evolution of goal-oriented systems. Our account draws on a broad range of disciplines, but we can only scratch the surface within each. We thus hope the present account elicits a collaborative enterprise, entailing both critical discussion as well as extensions in contexts we have missed.

Type
Target Article
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)