The primary objective in the present paper is to gain fundamental understanding of the performance achievable in ATM networks as a function of the various system characteristics. We derive limit theorems that characterize the achievable performance in terms of the offered traffic, the admissible region, and the revenue measure. The insights obtained allow for substantial simplifications in the design of real-time connection admission control algorithms. In particular, we describe how the boundaries of admissible regions with convex complements may be linearized - thus reducing the admissible region - so as to obtain a convenient loss network representation. The asymptotic results for the achievable performance suggest that the potential reduction in revenue is immaterial in high-capacity networks. Numerical experiments confirm that the actual reduction is typically negligible, even in networks of moderate capacity.

A filtering problem for a branching process is studied within the tree framework, and the minimal intensity of the splitting process is given.

The aim of this paper is to investigate the almost sure stability with a certain rate function λ(t) for a class of stochastic evolution equations in infinite dimensional spaces under various sufficient conditions. The results obtained here include exponential and polynomial stability as special cases. Much more refined sufficient conditions than the usual ones, for example, those described in [14], are obtained under our framework by the method of Liapunov functions. Two examples are given to illustrate our theory.

We investigate in this paper the stability of non-stationary stochastic processes, arising typically in applications of control. The setting is known as stochastic recursive sequences, which allows us to construct on one probability space stochastic processes that correspond to different initial states and even different control policies. It does not require any Markovian assumptions. A natural criterion for stability for such processes is that the influence of the initial state disappears after some finite time; in other words, starting from different initial states, the process will couple after some finite time to the same limiting (not necessarily stationary nor ergodic) stochastic process. We investigate this as well as other types of coupling, and present conditions for them to occur uniformly in some class of control policies. We then use the coupling results to establish new theoretical aspects in the theory of non-Markovian control.

This paper studies the expected average cost control problem for discrete-time Markov decision processes with denumerably infinite state spaces. A sequence of finite state space truncations is defined such that the average costs and average optimal policies in the sequence converge to the optimal average cost and an optimal policy in the original process. The theory is illustrated with several examples from the control of discrete-time queueing systems. Numerical results are discussed.

This work concerns controlled Markov chains with denumerable state space, (possibly) unbounded cost function, and an expected average cost criterion. Under a Lyapunov function condition, together with mild continuity-compactness assumptions, a simple necessary and sufficient criterion is given so that the relative value functions and differential costs produced by the value iteration scheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above.

We study a classical stochastic control problem arising in financial economics: to maximize expected logarithmic utility from terminal wealth and/or consumption. The novel feature of our work is that the portfolio is allowed to anticipate the future, i.e. the terminal values of the prices, or of the driving Brownian motion, are known to the investor, either exactly or with some uncertainty. Results on the finiteness of the value of the control problem are obtained in various setups, using techniques from the so-called enlargement of filtrations. When the value of the problem is finite, we compute it explicitly and exhibit an optimal portfolio in closed form.

Consider the optimal control problem of leaving an interval (– a, a) in a limited playing time. In the discrete-time problem, a is a positive integer and the player's position is given by a simple random walk on the integers with initial position x. At each time instant, the player chooses a coin from a control set where the probability of returning heads depends on the current position and the remaining amount of playing time, and the player is betting a unit value on the toss of the coin: heads returning +1 and tails − 1. We discuss the optimal strategy for this discrete-time game. In the continuous-time problem the player chooses infinitesimal mean and infinitesimal variance parameters from a control set which may depend upon the player's position. The problem is to find optimal mean and variance parameters that maximize the probability of leaving the interval [— a, a] within a finite time T > 0.

We explore a dynamic approach to the problems of call admission and resource allocation for communication networks with connections that are differentiated by their quality of service requirements. In a dynamic approach, the amount of spare resources is estimated on-line based on feedbacks from the network's quality of service monitoring mechanism. The schemes we propose remove the dependence on accurate traffic models and thus simplify the tasks of supplying traffic statistics required of network users. In this paper we present two dynamic algorithms. The objective of these algorithms is to find the minimum bandwidth necessary to satisfy a cell loss probability constraint at an asynchronous transfer mode (ATM) switch. We show that in both schemes the bandwidth chosen by the algorithm approaches the optimal value almost surely. Furthermore, in the second scheme, which determines the point closest to the optimal bandwidth from a finite number of choices, the expected learning time is finite.

The ‘Mabinogion sheep’ problem, originally due to D. Williams, is a nice illustration in discrete time of the martingale optimality principle and the use of local time in stochastic control. The use of singular controls involving local time is even more strikingly highlighted in the context of continuous time. This paper considers a class of diffusion versions of the discrete-time Mabinogion sheep problem. The stochastic version of the Bellman dynamic programming approach leads to a free boundary problem in each case. The most surprising feature in the continuous-time context is the existence of diffusion versions of the original discrete-time problem for which the optimal boundary is different from that in the discrete-time case; even when the optimal boundary is the same, the value functions can be very different.

The partially observed control problem is considered for stochastic processes with control entering into the diffusion and the observation. The maximum principle is proved for the partially observable optimal control. A pure probabilistic approach is used, and the adjoint processes are characterized as solutions of related backward stochastic differential equations in finite-dimensional spaces. Most of the derivation is identified with that of the completely observable case.

A continuous-time, non-linear filtering problem is considered in which both signal and observation processes are Markov chains. New finite-dimensional filters and smoothers are obtained for the state of the signal, for the number of jumps from one state to another, for the occupation time in any state of the signal, and for joint occupation times of the two processes. These estimates are then used in the expectation maximization algorithm to improve the parameters in the model. Consequently, our filters and model are adaptive, or self-tuning.

We present two sufficient conditions for detection of optimal and non-optimal actions in (ergodic) average-cost MDPs. They are easily interpreted and can be implemented as detection tests in both policy iteration and linear programming methods. An efficient implementation of a recent new policy iteration scheme is discussed.

A reference probability is explicitly constructed under which the signal and observation processes are independent. A simple, explicit recursive form is then obtained for the conditional density of the signal given the observations. Both non-linear and linear filters are considered, as well as two different information patterns.

This paper presents a state space and time discretization for the general average impulse control of piecewise deterministic Markov processes (PDPs). By combining several previous results we show that under some continuity, boundedness and compactness conditions on the parameters of the process, boundedness of the discretizations, and compactness of the state space, the discretized problem will converge uniformly to the original one. An application to optimal capacity expansion under uncertainty is given.

We investigate the impact of switching penalties on the nature of optimal scheduling policies for systems of parallel queues without arrivals. We study two types of switching penalties incurred when switching between queues: lump sum costs and time delays. Under the assumption that the service periods of jobs in a given queue possess the same distribution, we derive an index rule that defines an optimal policy. For switching penalties that depend on the particular nodes involved in a switch, we show that although an index rule is not optimal in general, there is an exhaustive service policy that is optimal.

The problem treated is that of controlling a process with values in [0, a]. The non-anticipative controls (µ(t), σ(t)) are selected from a set C(x) whenever X(t–) = x and the non-decreasing process A(t) is chosen by the controller subject to the condition where y is a constant representing the initial amount of fuel. The object is to maximize the probability that X(t) reaches a. The optimal process is determined when the function has a unique minimum on [0, a] and satisfies certain regularity conditions. The optimal process is a combination of ‘timid play' in which fuel is used gradually in the form of local time at 0, and ‘bold play' in which all the fuel is used at once.

The problem of estimating the transfer function of a linear system, together with the spectral density of an additive disturbance, is considered. The set of models used consists of linear rational transfer functions and the spectral densities are estimated from a finite-order autoregressive disturbance description. The true system and disturbance spectrum are, however, not necessarily of finite order. We investigate the properties of the estimates obtained as the number of observations tends to ∞ at the same time as the model order employed tends to ∞. It is shown that the estimates are strongly consistent and asymptotically normal, and an expression for the asymptotic variances is also given. The variance of the transfer function estimate at a certain frequency is related to the signal/noise ratio at that frequency and the model orders used, as well as the number of observations. The variance of the noise spectral estimate relates in a similar way to the squared value of the true spectrum.