The maximization of the long-term growth rate of expected utility is considered under drawdown constraints. In a general situation, the value and the optimal strategy of the problem are related to those of another ‘standard’ risk-sensitive-type portfolio optimization problem. Furthermore, an upside-chance maximization problem of a large deviation probability is stated as a ‘dual’ optimization problem. As an example, a ‘linear-quadratic’ model is studied in detail: the conditions to ensure the solvabilities of the problems are discussed and explicit expressions for the solutions are presented.

This paper is concerned with the analysis of Markov decision processes in which a natural form of termination ensures that the expected future costs are bounded, at least under some policies. Whereas most previous analyses have restricted attention to the case where the set of states is finite, this paper analyses the case where the set of states is not necessarily finite or even countable. It is shown that all the existence, uniqueness, and convergence results of the finite-state case hold when the set of states is a general Borel space, provided we make the additional assumption that the optimal value function is bounded below. We give a sufficient condition for the optimal value function to be bounded below which holds, in particular, if the set of states is countable.

In this paper we study discrete-time Markov decision processes with Borel state and action spaces. The criterion is to minimize average expected costs, and the costs may have neither upper nor lower bounds. We first provide two average optimality inequalities of opposing directions and give conditions for the existence of solutions to them. Then, using the two inequalities, we ensure the existence of an average optimal (deterministic) stationary policy under additional continuity-compactness assumptions. Our conditions are slightly weaker than those in the previous literature. Also, some new sufficient conditions for the existence of an average optimal stationary policy are imposed on the primitive data of the model. Moreover, our approach is slightly different from the well-known ‘optimality inequality approach’ widely used in Markov decision processes. Finally, we illustrate our results in two examples.

We consider a single-server queueing system at which customers arrive according to a Poisson process. The service times of the customers are independent and follow a Coxian distribution of order r. The system is subject to costs per unit time for holding a customer in the system. We give a closed-form expression for the average cost and the corresponding value function. The result can be used to derive nearly optimal policies in controlled queueing systems in which the service times are not necessarily Markovian, by performing a single step of policy iteration. We illustrate this in the model where a controller has to route to several single-server queues. Numerical experiments show that the improved policy has a close-to-optimal value.

Particle filters are Monte Carlo methods that aim to approximate the optimal filter of a partially observed Markov chain. In this paper, we study the case in which the transition kernel of the Markov chain depends on unknown parameters: we construct a particle filter for the simultaneous estimation of the parameter and the partially observed Markov chain (adaptive estimation) and we prove the convergence of this filter to the correct optimal filter, as time and the number of particles go to infinity. The filter presented here generalizes Del Moral's Monte Carlo particle filter.

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

We consider the problem of dynamic admission control in a Markovian loss system with two classes. Jobs arrive at the system in batches; each admitted job requires different service rates and brings different revenues depending on its class. We introduce the definition of a ‘preferred class’ for systems receiving mixed and single-class batches separately, and derive sufficient conditions for each system to have a preferred class. We also establish a monotonicity property of the optimal value functions, which reduces the number of possibly optimal actions.

In this note, we consider discrete-time finite Markov chains and assume that they are only partly observed. We obtain finite-dimensional normalized filters for basic statistics associated with such processes. Recursive equations for these filters are derived by means of simple computations involving conditional expectations. An application to the estimation of parameters of the so-called discrete-time batch Markovian arrival process is outlined.

In this paper, it is shown that the Foster-Lyapunov criterion is sufficient to ensure the existence of an invariant probability measure for both discrete- and continuous-time Markov processes without any additional hypotheses (such as irreducibility).

We study portfolio optimization problems in which the drift rate of the stock is Markov modulated and the driving factors cannot be observed by the investor. Using results from filter theory, we reduce this problem to one with complete observation. In the cases of logarithmic and power utility, we solve the problem explicitly with the help of stochastic control methods. It turns out that the value function is a classical solution of the corresponding Hamilton-Jacobi-Bellman equation. As a special case, we investigate the so-called Bayesian case, i.e. where the drift rate is unknown but does not change over time. In this case, we prove a number of interesting properties of the optimal portfolio strategy. In particular, using the likelihood-ratio ordering, we can compare the optimal investment in the case of observable drift rate to that in the case of unobservable drift rate. Thus, we also obtain the sign of the drift risk.

A multitype branching process is presented in the framework of marked trees and its structure is studied by applying the strong branching property. In particular, the Markov property and the expression for the generator are derived for the process whose components are the numbers of particles of each type. The filtering of the whole population, observing the number of particles of a given type, is discussed. Weak uniqueness for the filtering equation and a recursive structure for the linearized filtering equation are proved under a suitable assumption on the reproduction law.

We consider a heterogeneous population of identical particles divided into a finite number of classes according to their level of health. The partition can change over time, and a suitable exchangeability assumption is made to allow for having identical items of different types. The partition is not observed; we only observe the cardinality of a particular class. We discuss the problem of finding the conditional distribution of particle lifetimes, given such observations, using stochastic filtering techniques. In particular, a discrete-time approximation is given.

Stephens and Donnelly (2000) constructed an efficient sequential importance-sampling proposal distribution on coalescent histories of a sample of genes for computing the likelihood of a type configuration of genes in the sample. In the current paper a characterization of their importance-sampling proposal distribution is given in terms of the diffusion-process generator describing the distribution of the population gene frequencies. This characterization leads to a new technique for constructing importance-sampling algorithms in a much more general framework when the distribution of population gene frequencies follows a diffusion process, by approximating the generator of the process.

In this paper, we consider a failure point process related to the Markovian arrival process defined by Neuts. We show that it converges in distribution to a homogeneous Poisson process. This convergence takes place in the context of rare occurrences of failures. We also provide a convergence rate of the convergence in total variation of this point process using an approach developed by Kabanov, Liptser and Shiryaev for the doubly stochastic Poisson process driven by a finite Markov process.

De Iorio and Griffiths (2004) developed a new method of constructing sequential importance-sampling proposal distributions on coalescent histories of a sample of genes for computing the likelihood of a type configuration of genes in the sample by simulation. The method is based on approximating the diffusion-process generator describing the distribution of population gene frequencies, leading to an approximate sample distribution and finally to importance-sampling proposal distributions. This paper applies that method to construct an importance-sampling algorithm for computing the likelihood of samples of genes in subdivided population models. The importance-sampling technique of Stephens and Donnelly (2000) is thus extended to models with a Markov chain mutation mechanism between gene types and migration of genes between subpopulations. An algorithm for computing the likelihood of a sample configuration of genes from a subdivided population in an infinitely-many-alleles model of mutation is derived, extending Ewens's (1972) sampling formula in a single population. Likelihood calculation and ancestral inference in gene trees constructed from DNA sequences under the infinitely-many-sites model are also studied. The Griffiths-Tavaré method of likelihood calculation in gene trees of Bahlo and Griffiths (2000) is improved for subdivided populations.

We consider the dynamic scheduling of a multiclass queueing system with two servers, one dedicated (server 1) and one flexible (server 2), with no arrivals. Server 1 is dedicated to processing type-1 jobs while server 2 is primarily responsible for processing type-2 jobs but can also aid server 1 with its work. We address when it is optimal for server 2 to aid server 1 with type-1 jobs rather than process type-2 jobs. The objective is to minimize the total holding costs incurred until all jobs in the system are processed and leave the system. We show that the optimal policy can exhibit one of three possible structures: (i) an exhaustive policy for type-2 jobs, (ii) a nonincreasing switching curve in the number of type-1 jobs and (iii) a nondecreasing switching curve in the number of type-1 jobs. We characterize the necessary and sufficient conditions under which each policy will be optimal. We also explore the use of the optimal policy for the problem with no arrivals as a heuristic for the problem with dynamic arrivals.

We address the problem of tracking the time-varying linear subspaces (of a larger system) under a Bayesian framework. Variations in subspaces are treated as a piecewise-geodesic process on a complex Grassmann manifold and a Markov prior is imposed on it. This prior model, together with an observation model, gives rise to a hidden Markov model on a Grassmann manifold, and admits Bayesian inferences. A sequential Monte Carlo method is used for sampling from the time-varying posterior and the samples are used to estimate the underlying process. Simulation results are presented for principal subspace tracking in array signal processing.

We consider the stochastic sequence {Y t }t∈ℕ defined recursively by the linear relation Y t+1=A t Y t +B t in a random environment. The environment is described by the stochastic process {(A t ,B t )}t∈ℕ and is under the simultaneous control of several agents playing a discounted stochastic game. We formulate sufficient conditions on the game which ensure the existence of Nash equilibria in Markov strategies which have the additional property that, in equilibrium, the process {Y t }t∈ℕ converges in distribution to a stationary regime.

We study the multiserver queue with Poisson arrivals and identical independent servers with exponentially distributed service times. Customers arriving at the system are admitted or rejected according to a fixed threshold policy. Moreover, the system is subject to holding, waiting, and rejection costs. We give a closed-form expression for the average costs and the value function for this multiserver queue. The result will then be used in a single step of policy iteration in the model where a controller has to route to several finite-buffer queues with multiple servers. We numerically show that the improved policy has a close to optimal value.

We study linear jump parameter systems of differential and difference equations whose coefficients depend on the state of a semi-Markov process. We derive systems of equations for the first two moments of the random solutions of these jump parameter systems, and illustrate how moment equations can be used in examining their asymptotic stability.