2.1 Reinforcement learning

Reinforcement learning is an umbrella term for approaches to learning that bootstrap towards an optimal policy in a way that loops acting and planning. As will be seen, RL is a particularly useful framework for our question because it allows us to talk about reward which is not necessarily determined over the same state representation in which a decision takes place, in contrast to classical decision theory. That is, rewards are given as a scalar signal indexed only to a time, which allows them to have very little observable structure, whereas utility functions and preferences are assigned over outcomes, which are either themselves the objects of beliefs (Jeffrey Reference Jeffrey1990) or at least closely related to the objects of belief (Savage Reference Savage1972). This means that reward can be conceptualized in a way that is in principle separable from any particular cognitive representation of reality—and we can in turn ask how much reward itself, independent of state space representations, determines the agent’s values. Further, as an influential model of human decision-making which has already been incorporated into philosophical theorizing about cognition (see Haas (Reference Haas2022) for an overview), RL is a good fit for bridging rational choice and human cognition.

RL is typically defined in a Markov decision process (MDP).Footnote ¹ An MDP is a system where the environment is in a state $s \in S$ , and the agent chooses an act $a \in A$ , and can receive reward ${R_t}$ . The environment transitions between states according to a transition probability based on the current state and the chosen action: $P(s^{\rm{\prime}}|s,a)$ . This setting has the Markov property, meaning the state transition function is determined by the current state and act, and no past states or acts give any more information on top of the current ones. For instance, in the game of tic-tac-toe, the states could be game boards, the acts the choices available at each turn to the “O” player, and the state transition function the new position of the board after the move of the “X” player. The rewards would be a point given at the end for the “O” player winning, nothing for a draw, and a negative point for losing.

The agent is in charge of choosing how to act in each state (forming a policy $\pi $ ). Reinforcement learning is a family of methods for making a policy which are updated as the agent interacts with the environment, and in many conditions converge to the optimal policy, i.e., the one that maximizes (discounted, cumulative) reward (Sutton and Barto Reference Sutton and Barto2018).

Here is a basic RL algorithm, $Q$ -learning, which involves calculating a $Q$ -value for each act–state pair according to the following formula:

(1) $${Q_{{\rm{new}}}}\left( {{S_t},{A_t}} \right) = {Q_{{\rm{old}}}}\left( {{S_t},{A_t}} \right) + \alpha \left( {{R_{t+1}} + \gamma \mathop {{\rm{max}}}\limits_a Q\left( {{S_{t + 1}},a} \right) - {Q_{{\rm{old}}}}\left( {{S_t},{A_t}} \right)} \right).$$

Here, $\alpha $ is a learning rate and $\gamma $ is a discount factor. The formula tells us how to update the $Q$ -value for a (state, act) pair based on the old $Q$ -value, the reward received after taking the initial action, and the $Q$ -value of the best action in the next state. The $Q$ -learning formula is combined with a decision rule which tells the agent how to use the $Q$ -values to select actions. A basic rule is $\varepsilon $ -greedy, where the highest $Q$ -value act is selected with $1 - \varepsilon $ probability, and a random act is selected the rest of the time. The policy $\pi $ is defined as a mapping of states onto actions that the agent will select.

From this basic foundation, different versions of reinforcement learning can be articulated. On-policy methods like SARSA (state–action–reward–state–action) update values based on chosen acts, whereas the $Q$ -learning equation (1) updates based on the highest-valued subsequent act (these come apart when the agent picks an act at random). Model-based methods use knowledge of the environment to perform updates outside of the act–update loop described above, for instance by updating an entire future trajectory based on new information before acting.

For present purposes, the following features of RL are most significant. Reward is a quantity that is indexed to a time, and reflects what is of value in the environment, such as winning the game of tic-tac-toe. $Q$ -values are approximations of expected cumulative discounted reward that improve, and in some cases provably converge to optimality. Agents update $Q$ -values during action, and so do not need full information in order to plan: the updates that would have been intractable are split into smaller steps that involve (in the basic version) a single update per timestep.

2.2 State spaces

In this section, I’ll characterize state spaces and situate them within philosophical debates about choice. Classic problems of decision theory, such as Newcomb’s Problem, come with a predefined set of options. Before we begin thinking about the agent, we already have on the table a set of options: one box or two boxes, stay or switch, bet A or bet B. An option set, such as {one box, two boxes}, is a partition over acts, and it typically corresponds closely to another partition over possible states of the world, in this case {$1 in the opaque box, $1000 in the opaque box}.Footnote ² In cases like the Newcomb problem, it is natural to think about the set of options and states as given by the environment, since we don’t expect well-informed agents to think of the options differently.

However, many everyday problems we face do not have such an obvious structure. On the other end of the spectrum from a well-defined set of bets dependent on well-delineated states of the world, we have truly open-ended decisions like planning what to do tomorrow. When I am deciding what to do tomorrow, there are a wide range of available ways to lay out my options and the corresponding states. I could ask myself: should I get some work done or relax? In which case I might consider as relevant questions about states: am I too tired to write well? Will my friend be around for a soccer game? And so on. Further, I might start with states and then move to acts, for instance wondering whether it will rain or not and then thinking of options by how they fit with either scenario. These options and states form the landscape of deliberation: Levi (Reference Levi1990) makes the case that we need a notion of options that are on the table that does not reduce to choice-worthiness, since otherwise rational deliberation could not involve a narrowing of the options leading to choice. Given this openness, we can now see why a state space can often be fruitfully considered part of the agent rather than the environment. It is up to me whether I ask myself “should I work or relax?” or instead “should I stay home or go somewhere?” Further, this does not reduce to a notion of which distinctions I can draw or which concepts I have—while I cannot ask myself “should I play jai alai?” if I lack the concept of jai alai, I can easily fail to ask myself that question even when I have the concept. It seems that truly open-ended planning is typically a problem of too many possible questions rather than too few.

But many problems we face fall in between regimented bets and fully open planning. In deciding what to eat in a restaurant, the menu gives me a partition of acts but I might consider my own partition of states, such as: is the chef Filipino or are the vegetables pre-frozen? In approaching bets in a casino, I might merge multiple gambles into one, or classify outcomes by where they come in a sequence as pay-back or karma. In this sense, a “normal” decision has some open-ended elements, and some of the variation in human behavior can be explained by how different people employ state spaces.

To return to the RL setting, in an MDP we take a chunk of the real world and break it into abstract, typically discrete, units, i.e., states $S$ and acts $A$ . The state transition function $P$ can only be assigned over this partition, and likewise $Q\left( {{S_t},{A_t}} \right)$ assigns values to acts in states defined by the state space. Open-endedness occurs in the translation of reality into an MDP, and therefore in open-ended problems different agents face different MDPs which may have substantially different optimal strategies. In some applications, the designer of the agent makes this choice, while in others it is the agent herself.

Imagine a match played between two players who are friends with a long history of games, but where only one has played tic-tac-toe before. We can describe this situation as I did in the previous section just in terms of board position, and idealize the transition function as depending only on the current move and position. Even in tic-tac-toe, defining the state space is not obvious. In this case, a better model of the game might depend on the track records of the players, since players tend to be riskier when they are winning overall. Then the experienced player might want to think of the states not as board positions, but as board positions plus overall score. Since the novice player is still picking up on the rules, she might rely on perceptual cues and social signals such as a slight wince as her opponent notices a mistake she’s made. In this case the state space of the same match could be a rich three-dimensional scene linked to inferred mental states. This would explain why her choices depend on facial expressions and even tricks of lighting, unlike her opponent who only considers track record and board position. Even when the game is the same, different agents will approach it with a different state space, leading to different patterns of actions even when everything else about their mental states is the same.

In the case of Masha, which we started with, the questions she asks herself in planning, such as “should I have meat or dairy?,” articulate a state space. Just like the division in the case of tic-tac-toe, her planning about what to eat involves assigning values to acts in states, and forming a policy of what to choose when. Had she instead asked herself “should I have spicy food?,” she would be taking the same real-world problem of what to eat given her resources and tastes and modeling it according to a different partition.

In short, which state space an agent uses determines which problem she is solving. Ordinarily, we need to know which problem she is solving to tell what she should do or to predict what she will do on the assumption that she is rational. Or, in the formal case, we need to fit an MDP before solving it. The exception to this is when there is only one problem she should be solving, in which case we can either assume that’s what she’s doing and determine optimal behavior accordingly, or subject her to a quite different form of criticism about not seeing the problem the way she should. Thus, state space choice is a key ingredient of fitting rational models in cases with significant open-endedness, which are arguably the majority of our choices.

What about in ideal agents? Jeffrey’s (Reference Jeffrey1990) decision theory is famously partition invariant: by setting up decision problems over propositions, rather than the distinct types of acts and states, and calculating the desirability of a coarser proposition as a sum over the desirability of the finest-grain propositions under the coarser one, the desirability of an act is never altered by changing the way the space is divided.Footnote ³ Partition invariance does not hold in Savage’s (Reference Savage1972) theory, for instance, so one might argue that state space choice may or may not be relevant in a complete ideal theory.Footnote ⁴ As we’ll see, the importance of state space representations in valuing depends on features of the agent which are related to computational limitations, and so my argument does not apply to ideal agents.

Another issue with situating state spaces in the philosophical literature comes from an old debate about psychological realism. From a behaviorist perspective, state spaces are just a useful device to organize and predict the actions of agents. This could be said of any piece of the decision-theoretic model of the agent. But state spaces face an additional challenge. Thoma (Reference Thoma2021) brings out how revealed preference theory depends on taking for granted what the agent identifies as their options: in other words, we must be able to assume a particular state space in order to define the agent’s preferences in this behaviorist fashion. This, in turn, suggests that the open-ended choices in which state spaces are most usefully understood as operating is outside the purview of behaviorist theories. More generally, while we need not assume there is no way to provide a behaviorist characterization of state spaces, the most straightforward way to understand a state space is in realist terms, as the structure of the representations used in decision-making. In section 5, I present some psychological models of state spaces, which gives a fuller picture of which behaviors lead to state-space ascription.

I have been referring to the state space as if there is a single one used in each choice. This is a simplification I’ll use throughout this paper, which already seeks to complicate the classical picture of motivated choice. But it is worth noting that we likely use a set of different state space representations even in a single choice, for instance by shifting the question I ask myself when I get stuck about what to do. Models of choice in psychology often utilize structured state spaces, such as a hierarchical representation that joins state spaces at different levels of abstraction. Further complexity is added by considering that we sometimes reuse state spaces across problems, and other times build ad hoc representations for each problem.

A state space is the backbone of representations used in planning and choice. It is a partition of actions and states into units that are coarse-grained enough to enable efficient and general calculation. State spaces are implied in the structure of explicit questions like “should I go to the park or the beach?” but reflect features of cognition that may be pre-linguistic or implicit. While, in classic choice problems, a state space is assumed based on the environment alone, in a wide range of ordinary choices we are responsible for selecting our own state space. In order to argue that state space representations are a part of (human) valuing, I will now turn to setting up a simpler model of valuing based on RL.

6.1. The mere reflection approach

Divergence, on the simple RL theory, is a mere reflection of value. We can make this idea precise by considering the problem of making an RL agent. Imagine we are designing an agent to maximize cumulative reward in a set of environments. For instance, we might fix rewards as escaping a maze and minimizing travel time, and the environment as a probability distribution over a set of hedge mazes. Once we have fixed this question, we can think of the best state space using a value-of-representation (VOR) approach. This means defining the utility of a representation in decision-making as the expected utility of acting on the best plan made with that representation. Ho et al. (Reference Ho, Abel, Correa, Littman, Cohen and Griffiths2022) use exactly this approach to define a value of a construal $c$ , a simplified state space representation used in planning, over which we can define a policy ${\pi _c}$ . Each policy has a utility equivalent to the expected value of taking the acts suggested by the policy in the states specified by the construal:

(2) $$U({\pi_c}) = U(s_0) + {\sum\limits_a} {\pi_c} (a \ |\ {s_0}) {\sum\limits_{s^\prime}} P({s^\prime} \ |\ {s_0}, a) V{\phi_c} ({s^\prime}).$$

This gives us a measure of how good a policy is, and we could then immediately define the value of a representation as that of the best policy that could be built using that representation. But different representations will also differ as to their computational costs. Some are simpler, faster, and so forth than others, and so the full VOR is defined as

(3) $${\rm{VOR}}\left( c \right) = U\left( {\pi_c} \right) - C\left( c \right).$$

Intuitively, the best representation is the one that leads to taking acts with the highest expected utility given the costs of using that representation.

With this in mind, we can return to our maze-solving agent. Taking our set of hedge maze environments, which are three-dimensional and full of details, we could pick out a set of candidate state spaces that represent the environment and action types at a coarser grain. For instance, one family of state spaces would represent the maze as a $100 \times 100$ grid, and actions as moves in four directions from each cell. Another would render a three-dimensional environment but ignore color and texture. Amongst a suitable set of these, and given a measure of computational cost, we could calculate which state spaces have the highest VOR relative to the reward function already fixed (high reward for escape, small penalty for travel time). This would be the optimal state space (or set of state spaces) for this scenario of agent, environments, and reward.

I also want to note here that we could use the exact same machinery to diagnose a mismatch between a scenario and a state space, and to more generally infer backwards from state spaces to scenario ruled in and out as optimal given those state spaces. Further, we can do triangulations such as: given a state space, environment set, and agent properties, infer consistent and inconsistent reward functions. In other words, relative to a background context, the state space encodes latent value information in the form of constraints on which reward functions would be consistent with that state space. The more information we include in the context, such as environment set, cognitive constraints, and so on, the more latent reward information in the state space.

For example, across the same set of hedge mazes, we might design one agent to cut the hedges, one to navigate the mazes, and one to move obstacles out of the paths. Given these different goals, we might fix different reward signals and then use these together with the environment set and cognitive costs to specify a state space for each agent. The agent who trims the hedges will need to take different actions but also recognize different features of the environment: it would need to represent the length and height of the hedges. The navigating agent might get by with a very simplified representation that only represents differences in a two-dimensional floor plan or graphical representation of the maze layout. The obstacle-moving agent would need to represent the location and weight of the objects in the path, along with properties relevant to navigating in and out. The idea of latent reward information is just that given just these state spaces and none of the reward information, we could already guess which state space representations belong with which goals of the agents. This won’t be a one-to-one mapping by any means, but the more information we have about the environment and agent, the more constraints the state space imposes on possible reward signals which would render that state space optimal under the VOR measure.

For our hedge maze walker, then, the reward signal is prior to the latent value information. Prior both in terms of the order of hypothetical construction, and also prior in terms of information: the reward signal, plus the context, tells us everything we can know about the state space representation, since this is how it was chosen. The reverse will not be true except in special cases.Footnote ⁷ In this case, then, it seems completely uncontroversial to describe the explicit reward signal as the sole place where we find the agent’s values. So far, this story fits perfectly with explicit RL theories of value.

6.2 Mere reflection isn’t enough

We previously saw two senses in which the explicit rewards are prior in the hedge maze walker: causal and informational. Other things being equal, if both of these kinds of priority fail, we might then conclude that there is no longer a good reason to think of the latent value information as merely reflecting value rather than as making a distinct contribution to value. But we need to make that more precise.

Causal priority, in the case of the hedge maze walker, was priority in the narrative of creature construction, reflecting a kind of order of necessity or rationalization. We start with X and then add Y because X is part of the structure of the problem we find ourselves with, and Y is part of the solution. In fact, the reward signal does not need to be treated as prior in this sense: Abel et al. (Reference Abel, Dabney, Harutyunyan, Ho, Littman, Precup and Singh2021), for instance, take up the problem of starting with a desired behavior and then inferring reward structure that will help achieve those behaviors. For instance, when designing a robot vacuum, it would be a bit odd to start with a reward signal assigning value to states and times—instead, I most directly want the vacuum to perform a behavior: roam the room and collect all the dust. I can then think of reward signals as instrumental to generating these behaviors.

Suppose we concede that the reward signal is causally prior to the state space representation in the hedge maze walker. Under what conditions would the reward signal fail to be causally prior? In principle, the reward signal and state space representation might be totally independent, as is the case when I use a default state space representation such as a pixel-wise grid for representing a video game. In a biological creature, a state space might be learned from the environment in a way that does not depend on the agent’s explicit reward function, such as when I divide time by the days of the week simply because it is conventional in my society. However, in any natural or artificial creature where state spaces and reward signals are mutable, we might then have a subsequent kind of causal influence between the two even if they were originally independent. I might continue to use the days of the week representation because it aligns with my interests, or alter it to do so. In this case, the explicit reward signal can still be causally prior to the state space even if they each arose independently.

Informational priority has the same structure. Rather than consider relationships between these representations at an origin point, in realistic agents we will need to consider them as they evolve through learning and other kinds of change. The reward signal is informationally prior in the case of the hedge maze agent because it is maximally informative (given a scenario) about the state space without the reverse being true. In the original case, we imagined the state space chosen was a single optimal solution but in many cases there may be a large set of equally optimal representations. In that case, informational priority is consistent with not having enough information to get every property of the state space right—instead, the reward signal gets as close as any predictor could get. This type of informational priority is still very strong. In order to be charitable to the explicit reward view, then, we will consider informational priority to fail not whenever we can do better at predicting the state space given some further information, but only when this prediction is interpretable and the predictive value is substantial. This excludes cases where, for instance, I chose among equally good state spaces by using a nursery rhyme. In that case, the predictive value of knowing the nursery rhyme is marginal, and the relationship between the nursery rhyme and the chosen space is arbitrary or uninterpretable.

Now we can specify what is needed for both of these claims to fail, and thereby to undermine the rationale of treating the reward signal as merely reflected in the state space. First, examining the state space of an agent needs to give us tangible and significant constraints on what the agent must value, given a background scenario. Second, the state space cannot be derived from the reward signal and scenario either causally or informationally, either initially or through the process of development and learning. Instead, there must be another causal source for the state space, and the state space contain substantial information that can be meaningfully predicted not by the reward signal but by some other source in an interpretable way. Of course, this is precisely what we failed to identify in the case of human state spaces in section 5. So I conclude that the mere reflection view does not explain widespread features of human cognition.