In this chapter we introduce different representations and techniques for acting with nondeterministic models: nondeterministic state transition systems (Section 11.1), automata (Section 11.2), behavior trees (Section 11.3), and Petri nets (Section 11.4).

The development of more sophisticated and, especially, approximate sampling algorithms aimed at improving scalability in one or more of the senses already discussed in this book raises important considerations about how a suitable algorithm should be selected for a given task, how its tuning parameters should be determined, and how its convergence should be as- sessed. This chapter presents recent solutions to the above problems, whose starting point is to derive explicit upper bounds on an appropriate distance between the posterior and the approximation produced by MCMC. Further, we explain how these same tools can be adapted to provide powerful post-processing methods that can be used retrospectively to improve approximations produced using scalable MCMC.

In the past, techniques for natural language translation were not very relevant for acting and planning systems. However, with the recent advent of large language models and their various multimodal extensions into foundation models, this is no longer the case. This last part introduces large language models and their potential benefits in acting, planning, and learning. It discusses the perceiving, monitoring, and goal reasoning functions for deliberation.

Learning to act with probabilistic models is the area of reinforcement learning (RL), the topic of this chapter. RL in some ways parallels the adaptation mechanisms of natural beings to their environment, relying on feedback mechanisms and extending the homeostasis regulations to complex behaviors. With continual learning, an actor can cope with a continually changing environment.This chapter first introduces the main principles of reinforcement learning. It presents a simple Q-learning RL algorithm. It shows how to generalize a learned relation with a parametric representation. it introduces neural network methods, which play a major in learning and are needed for deep RL (Section 10.5) and policy-based RL (Section 10.6). The issues of aided reinforcement learning with shaped rewards, imitation learning, and inverse reinforcement learning are addressed next. Section 10.8 is about probabilistic planning and RL.

This chapter explores the benefits of non-reversible MCMC algorithms in improving sampling efficiency. Revisiting Hamiltonian Monte Carlo (HMC), the chapter discusses the advantages of breaking detailed balance and introduces lifting schemes as a tool to enhance exploration of the parameter space. It reviews non-reversible HMC and alternative algorithms like Gustafson’s method. The chapter also covers techniques like delayed rejection and the discrete bouncy particle sampler, offering a comparison between reversible and non-reversible methods. Theoretical insights and practical implementations are provided to highlight the efficiency gains from non-reversibility.

Chapter 2 examines how the use of “quantified self” as a shorthand for personal data necessarily indexes only one end, rather than the full spectrum, of technologists’ understanding of digitization and their own roles within it. Looking closely at the way digital executives talk about data in forums such as QS, among others, in fact reveals the contradictions, professional obfuscations, and hyperbole that continue to shape the self-tracking sector. Digital professionals may occasionally enfold concepts such as the "quantified self” into promotional “pitch theater” to stage self-monitoring devices as gadgets that produce faithful and objective data. My interactions with practitioners in these settings, however, point to the more varied social, legal, and fiscal advantages professionals reap from representing digital self-tracking and the data these devices produce as both plastic and precise. This chapter argues that the surface impression that technologists relate to data and modes of self-monitoring in reductive terms has to be weighed against ways executives pursue both digital ambiguity and objectivity as a meaningful corporate strategy.

To begin evaluating the interaction of “quantified self,” the concept, and Quantified Self (QS), the collective, with digital entrepreneurialism, it’s necessary to understand the influence of its originators, Kevin Kelly and Gary Wolf, on this construct’s form and function. Chapter 1 reviews how the two authors have coined the term and established the group as an expression of what Wolf has called the “culture of personal data” (Wolf, 2009). While the founders defer to the explanatory power of culture in situating the collective within the technological imaginary, this chapter examines how their own personal backgrounds as journalists and Wired magazine editors have shaped the semantic meaning of “quantified self” as a catchphrase that refers to the means and outputs of digital self-tracking and especially to QS as a community of technophiles. Although the role the forum has come to play within the commercial self-tracking sphere analyzed in this book does not fully align with its originators’ intentions, the framing they established has set the tone for many of the ways the collective has become socialized in the technological arena as well as how it has come to work within it.

This part of the book is devoted to acting, planning, and learning with operational models of actions expressed with a hierarchical task-oriented representation. Operational models are valuable for acting. They allow for detailed descriptions of complex actions handling dynamic environments with exogenous events. The representation relies on hierarchical refinement methods that describe alternative ways to handle tasks and react to events. A method can be any complex algorithm, decomposing a task into subtasks and primitive actions. Subtasks are refined recursively. Actions trigger the execution of sensory-motor procedures in closed loops that query and change the world stochastically.

Task and motion planning (TAMP) problems combine abstract causal relations from preconditions to effects with computational geometry, kinematics, and dynamics. This chapter is about the integration of planning for motion/manipulation with planning for abstract actions. It introduces the main sampling-based algorithms for motion planning. Manipulation planning is subsequently introduced. A few approaches specific to TAMP are then presented.

This chapter is about planning approaches with explicit time in the descriptive and operational models of actions, as well as in the models of the expected evolution of the world not caused by the actor. It describes a planning algorithm that handles durative and concurrent activities with respect to a predicted dynamics. Section 17.1 presents a knowledge representation for modeling actions and tasks with temporal variables using temporal refinement methods. Temporal plans and planning problems are defined as chronicles, i.e., collections of assertions and tasks with explicit temporal constraints. A planning algorithm with temporal refinement methods is developed in Section 17.2. The basic techniques for managing temporal and domain constraints are then presented in Section 17.3.

This chapter is about two key aspects of learning with deterministic models: learning heuristics to speed up the search for a solution plan and the automated synthesis of the model itself. We discuss how to learn heuristics for exploring parts of the search space that are more likely to lead to solutions. We then address the problem of how to learn a deterministic model, with a focus on learning action schemas.

Chapter 6 ultimately analyzes the Quantified Self (QS) as a gateway to the notions of difference that continue to shape the tech sector and therefore the devices that derive from it. As it considers the structural inequality that still constrains technological innovation, this chapter also analyzes QS as a site more specifically connected to the forms of privilege that impact how entrepreneurial extracurricular labor becomes converted into business advantage. It emphasizes that the modalities of participation that have rendered QS a community of tech acolytes unevenly regulate who can benefit from the group’s role as an instrument of professional transfiguration, connection, and access.

The camera slowly scans Chris Dancy’s face, first focusing on a profile of his bespectacled eyes, then quickly switching to a frontal shot to examine his contemplative expression at close range. Seconds later, the angle shifts again, the panorama now filmed as though from behind Dancy’s shoulder. The foreground looks blurry to start with. But once the lens adjusts, the viewer clearly sees the nearby cityscape at which Dancy longingly gazes.