Article contents
The value of uncertainty: An active inference perspective
Published online by Cambridge University Press: 19 March 2019
Abstract
We discuss how uncertainty underwrites exploration and epistemic foraging from the perspective of active inference: a generic scheme that places pragmatic (utility maximization) and epistemic (uncertainty minimization) imperatives on an equal footing – as primary determinants of proximal behavior. This formulation contextualizes the complementary motivational incentives for reward-related stimuli and environmental uncertainty, offering a normative treatment of their trade-off.
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © Cambridge University Press 2019
References
Attias, H. (2003) Planning by probabilistic inference. In: Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (AISTATS), Key West, Florida, January 3–6. https://dblp.uni-trier.de/rec/bibtex/conf/aistats/Attias03.Google Scholar
Baldassarre, G. & Mirolli, M., eds. (2013) Intrinsically motivated learning in natural and artificial systems. Springer.Google Scholar
Behrens, T. E. J., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. S. (2007) Learning the value of information in an uncertain world. Nature Neuroscience 10(9):1214–21. http://doi.org/10.1038/nn1954.Google Scholar
Berridge, K. C. (2004) Motivation concepts in behavioral neuroscience. Physiology & Behavior 81:179–209.Google Scholar
Botvinick, M. & Toussaint, M. (2012) Planning as inference. Trends in Cognitive Sciences 16:485–88. https://doi.org/10.1016/j.tics.2012.08.006.Google Scholar
Charnov, E. L. (1976b) Optimal foraging, the marginal value theorem. Theoretical Population Biology 9:129–36. doi: 10.1016/0040-5809(76)90040-X.Google Scholar
Christiansen, A. D., Mason, M. T. & Mitchell, T. M. (1991) Learning reliable manipulation strategies without initial physical models. Robotics and Autonomous Systems (Special Issue: Toward Learning Robots) 8:7–18. https://doi.org/10.1016/0921-8890(91)90011-9.Google Scholar
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. (2006) Cortical substrates for exploratory decisions in humans. Nature 441(7095):876–79. http://doi.org/10.1038/nature04766.Google Scholar
Dayan, P. & Sejnowski, T. J. (1996) Exploration bonuses and dual control. Machine Learning 25:5–22.Google Scholar
Donnarumma, F., Maisto, D. & Pezzulo, G. (2016) Problem solving as probabilistic inference with subgoaling: Explaining human successes and pitfalls in the tower of Hanoi. PLOS Computational Biology 12:e1004864. https://doi.org/10.1371/journal.pcbi.1004864Google Scholar
Friston, K. (2010) The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11:127–38. https://doi.org/10.1038/nrn2787Google Scholar
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O'Doherty, J. & Pezzulo, G. (2016a) Active inference and learning. Neuroscience & Biobehavioral Review 68:862–79. https://doi.org/10.1016/j.neubiorev.2016.06.022Google Scholar
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P. & Pezzulo, G. (2016b) Active inference: A process theory. Neural Computation 29:1–49. https://doi.org/10.1162/NECO_a_00912.Google Scholar
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T. & Pezzulo, G. (2015) Active inference and epistemic value. Cognitive Neuroscience 6(4):187–214. https://doi.org/10.1080/17588928.2015.1020053.Google Scholar
Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T. & Dolan, R. J. (2014) The anatomy of choice: Dopamine and decision-making. Philosophical Transactions of the Royal Society: Biological Sciences 369:20130481. https://doi.org/10.1098/rstb.2013.0481.Google Scholar
Friston, K. J., Lin, M., Frith, C. D., Pezzulo, G., Hobson, J. A. & Ondobaka, S. (2017) Active inference, curiosity and insight. Neural Computation 29(10):2633–2683. https://doi.org/10.1162/neco_a_00999.Google Scholar
Gallistel, C. R. & Gibbon, J. (2001) Computational versus associative models of simple conditioning. Current Directions in Psychological Science 10:146–50.Google Scholar
Gottlieb, J., Oudeyer, P.-Y., Lopes, M. & Baranes, A. (2013) Information-seeking, curiosity, and attention: Computational and neural mechanisms. Trends in Cognitive Sciences 17:585–93. https://doi.org/10.1016/j.tics.2013.09.001Google Scholar
Hayden, B. Y., Pearson, J. M. & Platt, M. L. (2011) Neuronal basis of sequential foraging decisions in a patchy environment. Nature Neuroscience 14:933–39. doi: 10.1038/nn.2856.Google Scholar
Inglis, I. R. (2000) The central role of uncertainty reduction in determining behaviour. Behaviour 137:1567–99. https://doi.org/10.1163/156853900502727Google Scholar
Inglis, I. R., Langton, S., Forkman, B. & Lazarus, J. (2001) An information primacy model of exploratory and foraging behaviour. Animal Behavior 62:543–57. https://doi.org/10.1006/anbe.2001.1780Google Scholar
Iodice, P., Ferrante, C., Brunetti, L., Cabib, S., Protasi, F., Walton, M. E. & Pezzulo, G. (2017) Fatigue modulates dopamine availability and promotes flexible choice reversals during decision making. Scientific Reports 7:535. https://doi.org/10.1038/s41598-017-00561-6.Google Scholar
Maisto, D., Donnarumma, F. & Pezzulo, G. (2015) Divide et impera: Subgoaling reduces the complexity of probabilistic inference and problem solving. Journal of the Royal Society Interface 12:20141335. https://doi.org/10.1098/rsif.2014.1335.Google Scholar
Oudeyer, P.-Y., Kaplan, F. & Hafner, V. (2007) Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11:265–86.Google Scholar
Pezzulo, G., Cartoni, E., Rigoli, F., Pio-Lopez, L. & Friston, K. (2016) Active inference, epistemic value, and vicarious trial and error. Learning & Memory 23:322–38. https://doi.org/10.1101/lm.041780.116Google Scholar
Pezzulo, G. & Rigoli, F. (2011) The value of foresight: How prospection affects decision-making. Frontiers in Neuroscience 5:79.Google Scholar
Pezzulo, G., Rigoli, F. & Chersi, F. (2013) The mixed instrumental controller: Using value of information to combine habitual choice and mental simulation. Frontiers in Cognition 4:92. https://doi.org/10.3389/fpsyg.2013.00092Google Scholar
Pezzulo, G., Rigoli, F. & Friston, K. J. (2015) Active inference, homeostatic regulation and adaptive behavioural control. Progress in Neurobiology 134:17–35.Google Scholar
Pezzulo, G., Rigoli, F. & Friston, K. J. (2018) Hierarchical active inference: A theory of motivated control. Trends in Cognitive Sciences 22(4):294–306. https://doi.org/10.1016/j.tics.2018.01.009.Google Scholar
Salamone, J. D., Correa, M., Farrar, A. M., Nunes, E. J. & Pardo, M. (2009) Dopamine, behavioral economics, and effort. Frontiers in Behavioral Neuroscience 3:13. https://doi.org/10.3389/neuro.08.013.2009.Google Scholar
Schmidhuber, J. (1991) Adaptive confidence and adaptive curiosity (No. FKI-149-91). Institut für Informatik, Technische Universitat.Google Scholar
Schwartenbeck, P., FitzGerald, T., Dolan, R. & Friston, K. (2013) Exploration, novelty, surprise, and free energy minimization. Frontiers in Psychology 4:710.Google Scholar
Singh, S., Barto, A.G. & Chentanez, N. (2005) Intrinsically motivated reinforcement learning. In: Advances in neural information processing systems, vol. 17, ed. Saul, L. K., Weiss, Y., & Bottou, L., pp. 1281–88. MIT Press.Google Scholar
Stoianov, I., Genovesio, A. & Pezzulo, G. (2015) Prefrontal goal codes emerge as latent states in probabilistic value learning. Journal of Cognitive Neuroscience 28:140–57.Google Scholar
Stoianov, I., Pennartz, C., Lansink, C., & Pezzulo, G. (2018) Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis. PLoS Computational Biology 14(9):e1006316. https://doi.org/10.1371/journal.pcbi.1006316.Google Scholar
Sutton, R. S. (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings of the Seventh International Conference on Machine Learning, ed. Porter, B. W. & Mooney, R. J., pp. 216–24. Morgan Kaufmann.Google Scholar
Walton, M., Kennerley, S., Bannerman, D., Phillips, P. & Rushworth, M. (2006) Weighing up the benefits of work: Behavioral and neural analyses of effort-related decision making. Neural Networks 19:1302–14. https://doi.org/10.1016/j.neunet.2006.03.005Google Scholar
- 9
- Cited by
Target article
How foraging works: Uncertainty magnifies food-seeking motivation
Related commentaries (22)
A neural basis for food foraging in obesity
Beyond uncertainty: A broader scope for “incentive hope” mechanisms and its implications
Beyond “incentive hope”: Information sampling and learning under reward uncertainty
Complex social ecology needs complex machineries of foraging
Considerations for the study of “incentive hope” and sign-tracking behaviors in humans
Does the “incentive hope” hypothesis explain food-wasting behavior among humans? Yes and no
Extending models of “How Foraging Works”: Uncertainty, controllability, and survivability
Food security and obesity: Can passerine foraging behavior inform explanations for human weight gain?
Food seeking and food sharing under uncertainty
Food-seeking behavior has complex evolutionary pressures in songbirds: Linking parental foraging to offspring sexual selection
Foraging extends beyond food: Hoarding of mental energy and information seeking in response to uncertainty
Hoarding all of the chips: Slot machine gambling and the foraging for coins
Hope, exploration, and equilibrated action schemes
How uncertainty begets hope: A model of adaptive and maladaptive seeking behavior
Mechanistic models must link the field and the lab
Overlapping neural systems underlying “incentive hope” and apprehension
Random isn't real: How the patchy distribution of ecological rewards may generate “incentive hope”
Simulating exploration versus exploitation in agent foraging under different environment uncertainties
The value of uncertainty: An active inference perspective
Unpredictable homeodynamic and ambient constraints on irrational decision making of aneural and neural foragers
“How Foraging Works”: Let's not forget the physiological mechanisms of energy balance
“Incentive hope” and the nature of impulsivity in low-socioeconomic-status individuals
Author response
Incentive hope: A default psychological response to multiple forms of uncertainty