In this book, we have studied computational reasoning principles and mechanisms to support choosing and performing actions.Here are some observations about the current status of work on those topics.

Extensive work has been done on automated planning, ranging from classical planning techniques to extended approaches dealing with temporal, hierarchical, nondeterministic, and probabilistic models. The field has progressed tremendously, and a strong community of scientists is continually producing new results, technology, and tools.

Issues related to acting have also attracted much attention, and the state of the art is broad and rich, but it is quite fragmented.The relationships among different approaches have not yet been studied in depth, and a unifying and formal account of acting is not available in the same way as it is in the field of automated planning.

Furthermore, the problems of how to generate plans and how to perform synthesized actions have been mainly studied separately, and a better understanding is needed of the relationships between planning and acting. One of the usual assumptions in research on planning is that actions are directly executable, and this assumption is used even in the work on interleaving online planning and execution. In most cases, however, acting cannot be reduced to the direct execution of atomic commands that have been chosen by a planner. Significant deliberation is needed for an actor to perform what is planned.

In this book, we have addressed the state of the art from a unifying perspective.We have presented techniques for doing planning with deterministic,hierarchical, temporal, nondeterministic, and probabilistic models, and have discussed approaches for reacting to events and refining actions into executable commands. In doing this, we have distinguished between two kinds of models:

1. • Descriptivemodels of actions specify the actor's “know what.”They describe which state or set of possible statesmay result from performing an action.They are used by the actor to reason about which actions may achieve its objectives.

2. • Operational models of actions specify the actor's “know how.” They describe how to perform an action, that is,what commands to execute in the current context and how to organize them to achieve the action's intended effects. The actor relies on operational models to perform the actions that it has decided to perform.

Over ten years ago, Malik Ghallab, Dana Nau, and Paolo Traverso gave us the first – and to date only – comprehensive textbook dedicated to the field of Automated Planning, providing a much needed resource for students, researchers and practitioners. Since then, this rich field has continued to evolve rapidly. There is now a unified understanding of what once seemed disparate work on classical planning. Models and methods to deal with time, resources, continuous change, multiple agents, and uncertainty have substantially matured. Cross-fertilization with other fields such as software verification, optimization, machine learning, and robotics has become the rule rather than the exception. A phenomenal range of applications could soon be within reach – given the right future emphasis for the field.

Today, the authors are back with a new book, Automated Planning and Acting. As the title indicates, this is not a mere second edition of the older book. In line with the authors’ analysis of where the future emphasis should lie for the field to realize its full impact, the book covers deliberative computational techniques for both planning and acting, that is, for deciding which actions to perform and also how to perform them. Automated Planning and Acting is more than a graduate textbook or a reference book. Not only do the authors outstandingly discharge their duties of educating the reader about the basics and much of the recent progress in the field, but they also propose a new framework from which the community can start to intensify research on deliberative acting and its integration with planning.

These aims are reflected in the book's content. The authors put the integration of planning and acting at the forefront by dedicating an entire chapter to a unified hierarchical model and refinement procedures that suit the needs of both planning and acting functions. Each chapter devoted to a particular class of representations also includes significant material on the integration of planning and acting using these representations. Overall, the book is more focused than its predecessor, and explores in even greater depth models and approaches motivated by the needs of planning and acting in the real world, such as handling time and uncertainty. At the same time, the authors successfully balance breadth and depth by providing an elegant, concise synthesis of a larger body of work than in their earlier text.

Chapter 2 concentrated mostly on planning with descriptive action models. Although it described some ways for an actor to receive guidance from such a planner, it did not describe the operational models that an actor might need to perform the planned actions. In the current chapter, we present a formalism for operational models and describe how to use these models for deliberative acting.

Section 3.1 describes a formalism for operational models based on refinement methods. A method specifies how to accomplish a task (an abstract activity of some kind) by refining it into other activities that are less abstract. These activities may include other tasks that will need further refinement and commands that can be sent to the execution platform. Section 3.2 describes an acting procedure, RAE, that uses a collection of refinement methods to generate and traverse a refinement tree similar to the one in Figure 1.2. It recursively refines abstract activities into less abstract activities, ultimately producing commands to the execution platform.

If we modify the refinement methods by replacing the commands with descriptive models, the modified methods can also be used for planning. The basic idea is to augment the acting procedure with predictive lookahead of the possible outcome of commands that can be chosen. Section 3.3 describes a planner, SeRPE, that does this. Section 3.4 describes how to integrate such a planner into acting procedures.

Although the formalism in this chapter removes many of the simplifying assumptions that we made in Chapter 2, it still incorporates some assumptions that do not always hold in practical applications. Section 3.5 discusses these and also includes historical remarks.

OPERATIONAL MODELS

In this section, we present a formalism for operational models of actions and describe how to use these models for deliberative acting. This formalism weakens or removes several of the simplifying assumptions that we made in Section 2.1.1:

1. • Dynamic environment. The environment is not necessarily static.Our operational models deal with exogenous events, that is,events due to other causes than the actor's actions.

2. • Imperfect information. In Section 2.1.1, we assumed that the actor had perfect information about its environment. In reality, it is rare for an actor to be able to know the current value of every state variable and to maintain this knowledge while the world evolves.

This book is about methods and techniques that a computational agent can use for deliberative planning and acting, that is, for deciding both which actions to perform and how to perform them, to achieve

some objective. The study of deliberation has several scientific and engineering motivations.

Understanding deliberation is an objective for most cognitive sciences. In artificial intelligence research, this is done by modeling deliberation through computational approaches both to enable it and to allow it to be explained. Furthermore, the investigated capabilities are better understood by mapping concepts and theories into designed systems and experiments to test empirically,measure,and qualify the proposed models.

The engineering motivation for studying deliberation is to build systems that exhibit deliberation capabilities and develop technologies that address socially useful needs.A technological system needs deliberation capabilities if it must autonomously perform a set of tasks that are too diverse – ormust be done in environments that are too diverse – to engineer those tasks into innate behaviors. Autonomy and diversity of tasks and environments is a critical feature in many applications, including robotics (e.g., service and personal robots; rescue and exploration robots; autonomous space stations, satellites, or vehicles), complex simulation systems (e.g., tutoring, training or entertainment), or complex infrastructure management (e.g., industrial or energy plants, transportation networks, urban facilities).

MOTIVATION AND COVERAGE

The coverage of this book derives from the view we advocated for in our previous work [231], which we now briefly summarize.

Automated planning is a rich technical field, which benefits from the work of an active and growing research community. Some areas in this field are extensively explored and correspond to a number of already mature techniques. However, there are other areas in which further investigation is critically needed if automated planning is to have a wider impact on a broader set of applications. One of the most important such areas, in our view is the integration of planning and acting.This book covers several different kinds of models and approaches – deterministic, hierarchical, temporal,nondeterministic and probabilistic – and for each of them, we discuss not only the techniques themselves but also how to use them in the integration of planning and acting.

In this chapter, we explore various approaches for using probabilistic models to handle the uncertainty and nondeterminism in planning and acting problems. These approaches are mostly based on dynamic programming optimization methods for Markov decision processes.We explain the basic principles and develop heuristic search algorithms for stochastic shortest-path problems. We also propose several sampling algorithms for online probabilistic planning and discuss how to augment with probabilistic models refinement methods for acting. The chapter also discusses the critical issue of specifying a domain with probabilistic models.

Our motivations for using probabilistic models, and our main assumptions, are briefly introduced next. Section 6.2 defines stochastic shortest-path problems and basic approaches for solving them. Different heuristic search algorithms for these problems are presented and analyzed in Section 6.3.Online probabilistic planning approaches are covered in Section 6.4.Refinement methods for actingwith probabilisticmodels are presented in Section 6.5. Sections 6.6 and 6.7 are devoted to factored representations and domain modeling issues with probabilistic models, respectively. The main references are given in the discussion and historical remarks Section 6.8. The chapter ends with exercises.

INTRODUCTION

Some of the motivations for deliberation with probabilistic models are similar to those introduced in Chapter 5 for addressing nondeterminism: the future is never entirely predictable, models are necessarily incomplete, and, even in predictable environments, complete deterministic models are often too complex and costly to develop. In addition, probabilistic planning considers that the possible outcomes of an action are not equally likely. Sometimes, one is able to estimate the likelihood of each outcome, relying for example on statistics of past observations. Probabilistic planning addresses those cases in which it is desirable to seek plans optimized with respect to the estimated likelihood of the effects of their actions.

The usual formal model of probabilistic planning is that of Markov decision processes (MDPs). An MDP is a nondeterministic state-transition system together with a probability distribution and a cost distribution.The probability distribution defines how likely it is to get to a state s' when an action a is performed in a state s.

A probabilistic state-transition system is said to be Markovian if the probability distribution of the next state depends only on the current state and not on the sequence of states that preceded it.

This chapter is about planning and acting approaches in which time is explicit in the descriptive and operational models of actions, as well as in the models of the expected evolution of the world. It describes several algorithms and computation methods for handling durative and concurrent activities with respect to a predicted dynamics.

The first section addresses the need of making time explicit in the deliberation of an actor. A knowledge representation for modeling actions with temporal variables is presented in Section 4.2. It relies on an extension of the refinement methods introduced earlier, which are seen here as chronicles, that is, collections of assertions and tasks with explicit temporal constraints.A planning algorithm with temporal refinement methods is developed in Section 4.3.The basic techniques formanaging temporal constraints and the controllability of temporal plans are then presented in Section 4.4.Acting problems with temporal domain models are discussed, considering different types of operational models, in Section 4.5. The chapter concludes with a discussion and historical remarks, followed by exercises.

INTRODUCTION

To perform an action, different kinds of resources may need to be borrowed (e.g., space, tools) or consumed (e.g., energy). Time is a resource required by every action, but it differs from other types of resources. It flows independently from whether any actions are being performed, and it can be shared ad infinitum by independent actors as long as their actions do not interfere with each other.

In previous chapters, we left time implicit in our models: an action produced an instantaneous transition from one state to the next.However, deliberative acting often requires explicit temporal models of actions.Rather than just specifying an action's preconditions and effects, temporal modelsmust specify what things an action requires and what events it will cause at various points during the action's performance. For example, moving a robot r 1 from a loading dock d 1 to a loading dock d 2 does not require d 2's availability at the outset but does require it shortly before r 1 reaches d 2.

Actions may, and sometimes must, overlap, even if their conditions and effects are not independent. As one example, r 1 may move from d 1 to d 2 while r 2 is moving from d 2 to d 1.As another, opening a door that has a knob and a spring latch that controls the knob requires two tightly synchronized actions: (i) pushing and maintaining the latch while (ii) turning the knob.

This appendix provides background information about several of the search algorithms used in this book. These are nondeterministic state-space search (Section A.1) and And/Or search (Section A.2).

NONDETERMINISTIC STATE-SPACE SEARCH

Many of the planning algorithms in this book have been presented as nondeterministic search algorithms and can be described as instances of Algorithm A.1, Nondeterministic-Search. In most implementations of these algorithms, line (iii) corresponds to trying several members of R sequentially in a trial-and-error fashion. The “nondeterministically choose” command is an abstraction that lets us ignore the precise order in which those values are tried. This enables us to discuss properties that are shared by a wide variety of algorithms that search the same space of partial solutions, even though those algorithms may visit different nodes of that space in different orders.

There are several theoretical models of nondeterministic choice that are more-orless equivalent mathematically [213, 464, 131]. The one that is most relevant for our purposes is the nondeterministic Turing machine model,which works roughly as follows.

Let ψ(P) be a process produced by calling Nondeterministic-Search on a search problem P. Whenever this process reaches line (iii), it replaces ψ(P) with |R| copies of ψ(P) running in parallel: one copy for each r ∈ R. Each process corresponds to a different execution trace of ψ(P), and each execution trace follows one of the paths in ψ(P)'s search tree (see Figure A.1). Each execution trace that terminates will either return failure or return a purported answer to P.

Two desirable properties for a search algorithm ψ are soundness and completeness, which are defined as follows:

1. • ψ is sound over a set of search problems P if for every P ∈ P and every execution trace of ψ(P), if the trace terminates and returns a value π ≠ failure, then π is a solution for P. This will happen if the solution test in line (i) is sound.

Algorithm A.1 Equivalent iterative and recursive versions of a generic nondeterministic search algorithm. The arguments include the search problem P and (in the recursive version) a partial solution π, the initial value of which should be the empty plan.

In this chapter we drop the unrealistic assumption of determinism, that is, the assumption that each action performed in one state leads deterministically to one state. This apparently simple extension introduces uncertainty in the model of the domain and requires new approaches to planning and acting. Deliberation must take into account that actions can lead to a set of states; plans are no longer sequences of actions, but conditional plans; solutions may have different strengths. Deliberative acting with nondeterministic models allows us to take into account uncertainty when actions are performed.

The main motivations for planning and acting with nondeterministic models are in Section 5.1. The planning problem is formalized in Section 5.2. In the subsequent three sections we present some different approaches to planning with nondeterministic models: And/Or graph search (Section 5.3), symbolic model checking (Section 5.4), and determinization techniques (Section 5.5). In Section 5.6,we present techniques that interleave planning and acting. In Section 5.7, we present planning techniques with refinement methods and nondeterministic models, and in Section 5.8 we show techniques for deliberative acting with input/output automata. Comparisons among different approaches and main references are given in the discussion and historical remarks in Section 5.9. The chapter ends with a few exercises.

INTRODUCTION AND MOTIVATION

Recall that in deterministic models, the prediction of the effects of an action is deterministic: only one state is predicted as the result of performing an action in a state (seeChapter 2, Section 2.1.1, assumption in item 3). Nondeterministic models predict alternative options: an action when applied in a state may result in one among several possible states. Formally, γ (s, a) returns a set of states rather than a single state. The extension allowed by nondeterministic models is important because it allows for modeling the uncertainty of the real world.

In some cases, using a deterministic or a nondeterministic model is a design choice. For instance, in the real world, the execution of an action may either succeed or fail. Despite this, in many cases, it still makes sense to model just the so-called nominal case (in which failure does not occur), monitor execution, detect failure when it occurs, and recover, for example, by replanning or by re-acting with some failure-recovery mechanism. In these cases, deterministic models can still be a convenient choice.

This chapter introduces informally the concepts and technical material developed in the rest of the book. It discusses in particular the notion of deliberation, which is at the core of the interaction between planning and acting. Section 1.1 motivates our study of deliberation from a computational viewpoint and delineates the scope of the book. We then introduce a conceptual view of an artificial entity, called an actor, capable of acting deliberately on its environment, and discuss our main assumptions.Deliberation models and functions are presented next. Section 1.4 describes two application domains that will be simplified into illustrative examples of the techniques covered in rest of the book.

PURPOSE AND MOTIVATIONS

First Intuition

What is deliberative acting? That is the question we are studying in this book.We address it by investigating the computational reasoning principles and mechanisms supporting how to choose and perform actions.

We use the word action to refer to something that an agent does, such as exerting a force, a motion, a perception or a communication, in order to make a change in its environment and own state. An agent is any entity capable of interacting with its environment. An agent acting deliberately is motivated by some intended objective. It performs one or several actions that are justifiable by sound reasoning with respect to this objective.

Deliberation for acting consists of deciding which actions to undertake and how to perform them to achieve an objective. It refers to a reasoning process, both before and during acting, that addresses questions such as the following:

1. • If an agent performs an action, what will the result be?

2. • Which actions should an agent undertake,and how should the agent perform the chosen actions to produce a desired effect?

Such reasoning allows the agent to predict, to decide what to do and how do it, and to combine several actions that contribute jointly to the objective. The reasoning consists in using predictive models of the agent's environment and capabilities to simulate what will happen if the agent performs an action. Let us illustrate these abstract notions intuitively.