Consequences and attributes; introductory ideas

We have shown that two ingredients are combined in our decision-making: probability and utility. But utility has not been well defined. Up to this point, we have given detail of the probabilistic part of the reasoning. We now present the basic theory with regard to utility. Probabilities represent our opinions on the relative likelihood of what is possible, but not certain; utilities encode our feelings regarding the relative desirability of the various consequences of our decisions. Examples of where utilities fit into the decision-making process were given in Section 1.2. A special case is expected monetary value (EMV), dealt with in Section 3.5.2.

There can be various components of utility, depending on the application. These components will be referred to as ‘attributes’. For example, in a study of airport development for Mexico City, de Neufville and Keeney (1972) considered attributes of the following kind.

• cost

• airport capacity

• safety

• access time

• noise

• displacement of people.

There will usually be a set of such attributes, which depends on the problem at hand. Attributes may have

• increasing value, for example money, profit, volume of sales, time saved in travelling, and wilderness areas available for enjoyment

• decreasing value, for example size of an oil spill, level of noise pollution, and deaths and injuries in an accident.

Introduction

To solve a static, constrained optimization problem, we typically employ the Lagrange multiplier method. The solution to the problem is characterized by the first-order conditions of the Lagrange problem. To solve a stochastic, intertemporal optimization problem, the optimal control policy is characterized by the first-order conditions of the Bellman equation. In this chapter we shall introduce this method of dynamic optimization under uncertainty. One of the objectives is to make the reader feel as comfortable using the Bellman equation in dynamic models as using the Lagrange multiplier method in static models.

The chapter begins with a one-sector optimal growth model. We go through the derivation of the corresponding Bellman equation step by step to convey the mathematical reasoning behind this powerful tool using a real economic problem, not an abstract mathematical formulation. Then we examine the mathematical structure of the stochastic optimization problem, including the existence of the optimal control, the differentiability of the value function, the transversality condition, and the verification theorem. More importantly, we summarize the Bellman equation in a cookbook fashion to make it easy to use. To illustrate, we apply the Bellman equation to several well-known models, such as portfolio selection, index bonds, exhaustible resources, adjustment costs, and life insurance. To make the presentation self-contained, we provide a brief introduction to each topic.

Then, we extend the method from time-additive utility functions with a constant discount rate to a class of recursive utility functions.

This is an introduction to stochastic control theory with applications to economics. There are many texts on this mathematical subject; however, most of them are written for students in mathematics or in finance. For those who are interested in the relevance and applications of this powerful mathematical machinery to economics, there must be a thorough and concise resource for learning. This book is designed for that purpose. The mathematical methods are discussed intuitively whenever possible and illustrated with many economic examples. More importantly, the mathematical concepts are introduced in language and terminology familiar to first-year graduate students in economics.

The book is, therefore, at a second-year graduate level. The first part covers the basic elements of stochastic calculus. Chapter 1 is a brief review of probability theory focusing on the mathematical structure of the information set at time t, and the concept of conditional expectations. Many theorems related to conditional expectations are explained intuitively without formal proofs.

Chapter 2 is devoted to the Wiener process with emphasis on its irregularities. The Wiener process is an essential component of modeling shocks in continuous time. We introduce this important concept via three different approaches: as a limit of random walks, as a Markov process with a specific transition probability, and as a formal mathematical definition which enables us to derive and verify variants of the Wiener process. The best way to understand the irregularities of the Wiener process is to examine its sample paths closely.

Introduction

We have shown in Chapter 4 that the Bellman equation enables us to derive the equations that govern the optimal control policies. However, to have a better understanding of these behavioral functions we often need to know the functional form of the value function. For example, if the indirect utility function is of constant relative risk aversion in Merton's consumption and portfolio model, then we have shown that the share of wealth invested in each risky asset is constant over time. In this chapter we study the methods that determine the functional form of the value function of a stochastic, intertemporal optimization problem.

By definition, the value function depends on the specification of the objective function and the underlying controlled diffusion process. Changing the objective function or the controlled diffusion process would change the functional form of the value function and, hence, the optimal control.

We shall divide economic problems into four different classes of problems. The first class is the one in which the diffusion equation is linear in both state and control variables and the objective function is quadratic. This is the so-called linear–quadratic problem in control theory. The second class is the one in which the controlled diffusion process is linear in both state and control variables and the objective function exhibits hyperbolic absolute risk aversion (HARA). This class of functions contains most of the commonly employed objective functions and therefore deserves a special mention.

Introduction

In this chapter we introduce the concept and the major properties of a Wiener process. This stochastic process is essential in building the theory of stochastic optimization in continuous time. Unlike Chapter 1, we provide proofs for most of the theorems, because going through the proofs will enhance our understanding of the process under study.

We begin with a heuristic approach to the Wiener process. Specifically, the Wiener process can be generated as the limiting process of the random walk by letting the time interval go to zero. Then we introduce Markov processes, a subject familiar to economists. The purpose is to show that a Wiener process is a Markov process with a normally distributed transition probability. That gives us another perspective on Wiener processes. Finally, we formally define a Wiener process and show its major properties, using a measure-theoretic approach. Section 2.4.1 shows various ways to generate more Wiener processes from a given one. We stress that the understanding of the zero set is crucial to have a good grasp of this special stochastic process. We also stress that the sample path of a Wiener process is everywhere continuous and nowhere differentiable.

A Heuristic Approach

From Random Walks to Wiener Process

Recall that a random walk is a stochastic process such that, at each time interval Δt, it takes a step forward with probability p, and a step backward with probability 1 – p, and such that all steps are mutually independent.

Introduction

In this chapter we shall discuss some issues related to the boundaries of a stochastic optimization problem. We begin with the important issue of the nonnegativity constraint, which is usually overlooked in the literature. Mathematically, the controlled stochastic differential equation that represents the law of motion is defined on the whole real line. The solution to such an equation cannot rule out the possibility that the state variables and/or the control variables are negative at some times on a set of positive probability. Economically, these variables typically represent consumption, stocks of capital, capital–labor ratio, or exhaustible resources. It makes no sense to have negative values for these variables. Since the nonnegativity constraint is not part of the mathematical solutions, more work needs to be done.

To address this issue, we use the optimal growth problem as our example. Our question is this: Is it possible that, with positive probability, the solution to the stochastic Solow equation explodes in finite time or has a negative capital–labor ratio at some point in time? We point out the major difficulties of this problem, and then discuss some methods to tackle it. Among them, we include the comparison theorem, which enables us to compare the solution of one differential equation with the solution of another differential equation that has some nice properties. Then we introduce a reflection method developed by Chang and Malliaris (1987) to solve this nonnegativity problem of optimal growth.

The theme of this chapter is control theory. We discuss what it means to say that a linear system is stable, and then present some of the themes of H∞ control theory, presented from an operator-theoretic point of view.

One of the main aims of modern control theory is to achieve robustness, that is, the stabilization of a system subject to perturbations, measurement errors, and the like. In order to study this we require a measure of the distance between systems, and it turns out that the operator gap is the “correct” one to use. Another way of measuring distances, the so-called chordal metric between meromorphic functions, turns out to be closely related.

Stability theory

The basic signal spaces in this chapter are vector-valued L2(0, ∞) or ℓ2(ℤ+) spaces, and we are concerned with shift-invariant input–output operators T. Our first result shows that, if the domain of such an operator is the whole space, then it is necessarily bounded (a result in automatic continuity theory).

Theorem 4.1.1Let T: L2(0, ∞; ℂm) → L2(0, ∞; ℂp) be an operator commuting with the right shift Rλfor some λ > 0. Then T is bounded.

Proof: It is sufficient to prove the result for m = p = 1, since in general T may be represented by a p × m matrix of shift-invariant operators from L2(0, ∞) to itself.

So far we have worked almost entirely with signal spaces of the form ℓ2(ℤ+) or L2(0, ∞) and their full-axis analogues; in physical terms, these are spaces of finite-energy signals, which die away in some sense at infinity. In this chapter we shall work with what may loosely be described as finite-power signals or, still more loosely, as persistent signals.

Persistent signals include classes of signals with some regularity properties, such as periodic and almost-periodic signals, as well as much more general spaces of signals in which the notion of “power” is less clearly defined. In particular, we are are able to discuss concepts such as the idea of a white noise signal in a rigorous and largely non-stochastic framework. Persistent signals in general can be taken as the inputs and outputs of linear systems (the term filter is commonly used here), as we shall see.

Almost-periodic functions

Almost-periodic functions defined on the real line form a class of functions that has been much studied since the 1920s. Our aim in this section is to derive their fundamental properties and to bring out their similarities with the theory of periodic functions.

It should be emphasised at the start that this book does not claim to be an exhaustive treatise on either linear operators or linear systems, but it presents an introduction to the common ground between the two subjects, one pure mathematical and one applied, by regarding a linear system as a (causal) shift-invariant operator on a Hilbert space such as ℓ2(ℤ+) or L2(0, ∞). It therefore includes material on Hardy spaces, shift-invariant operators, the commutant lifting theorem, and almost-periodic functions, which might traditionally be regarded as “pure” mathematics, and is suitable for those working in analysis who wish to learn more advanced material on linear operators.

At the same time, it is hoped that students and researchers in systems and control will find the approach taken attractive, including as it does much recent material on the mathematical side of systems theory, which cannot easily be found elsewhere: these include recent developments in robust control, power signal spaces, and the input–output approach to time-delay systems. Parts of this book have been expounded in graduate courses and other lectures at that level and could be used for a similar purpose elsewhere.

Chapter 1 begins with a review of basic operator theory without proofs. All this material can be found in any introductory course and many textbooks, and so is included mostly for reference. The other main topic of this chapter, which is treated in considerably more detail, is that of Hardy spaces, which are Banach spaces of analytic functions on the disc or half-plane.