Introduction

Machine learning refers to the design of computer algorithms for gaining new knowledge, improving existing knowledge, and making predictions or decisions based on empirical data. Applications of machine learning include speech recognition [164, 275], image recognition [60, 110], medical diagnosis [309], language understanding [50], biological sequence analysis [85], and many other fields. The most important requirement for an algorithm in machine learning is its ability to make accurate predictions or correct decisions when presented with instances or data not seen before.

Classification of data is a common task in machine learning. It consists of finding a function z = G(y) that assigns to each data sample y its class label z. If the range of the function is discrete, it is called a classifier, otherwise it is called a regression function. For each class label z, we can define the acceptance region Az such that y ∈ Az if and only if z = G(y). An error occurs if the classifier assigns a wrong class to y. The probability of classification error

is called the generalization error in machine learning, where Z denotes the actual class to which the observation variable Y belongs. The classifier that minimizes the generalization error is called the Bayes classifier and the minimized ε(G) is called the Bayes error. In practical applications, we generally do not know the probability distribution of (Y, Z).

In this chapter we will study statistical methods to estimate parameters and procedures to test the goodness of fit of a model to the experimental data. We are primarily concerned with computational algorithms for these methods and procedures. The expectation-maximization (EM) algorithm for maximum-likelihood estimation is discussed in detail.

Classical numerical methods for estimation

As we stated earlier, it is often the case that a maximum-likelihood estimate (MLE) cannot be found analytically. Thus, numerical methods for computing the MLE are important. Finding the maximum of a likelihood function is an optimization problem. There are a number of optimization algorithms and software packages. In this and the next sections we will discuss several important methods that are pertinent to maximization of a likelihood function: the method of moments, the minimum χ2method, the minimum Kullback–Leibler divergence method, and the Newton–Raphson algorithm. In Section 19.2 we give a full account of the EM algorithm, because of its rather recent development and its increasing applications in signal processing and other science and engineering fields.

Method of moments

This method is typically used to estimate unknown parameters of a distribution function by equating the sample mean, sample variance, and other higher moments calculated from data to the corresponding moments expressed in the parameters of interest.

In this chapter we will discuss some important inequalities used in probability and statistics and their applications. They include the Cauchy–Schwarz inequality, Jensen's inequality, Markov and Chebyshev inequalities. We then discuss Chernoff's bounds, followed by an introduction to large deviation theory.

Inequalities frequently used in probability theory

Cauchy–Schwarz inequality

The Cauchy–Schwarz inequality is perhaps the most frequently used inequality in many branches of mathematics, including linear algebra, analysis, and probability theory. In engineering applications, a matched filter and correlation receiver are derived from this inequality. Since the Cauchy–Schwarz inequality holds for a general inner product space, we briefly review its properties and in particular the notion of orthogonality. We assume that the reader is familiar with the notion of field and vector space (e.g., see Birkhoff and MacLane [28] and Hoffman and Kunze [153]). Briefly stated, a field is an algebraic structure with notions of addition, subtraction, multiplication, and division, satisfying certain axioms. The most commonly used fields are the field of real numbers, the field of complex numbers, and the field of rational numbers, but there is also a finite field, known as a Galois field. Any field may be used as the scalars for a vector space.

The estimation of a random variable or process by observing other random variables or processes is an important problem in communications, signal processing and other science and engineering applications. In Chapter 18 we considered a partially observable RVX = (Y, Z), where the unobservable part Z is called a latent variable. In this chapter we study the problem of estimating the unobserved part using samples of the observed part. In Chapter 18 we also considered the problem of estimating RVs, called random parameters using the maximum a posteriori probability (MAP) estimation procedure. When the prior distribution of the random parameter is unknown, we normally assume a uniform distribution, and then the MAP estimate reduces to the maximumlikelihood estimate (MLE) (see Section 18.1.2): if the prior density is not uniform, the MLE is not optimal and does not possess the nice properties described in Section 18.1.2. If an estimator θ = T(X) has a Gaussian distribution N(µ, ∑), its log-likelihood function is a quadratic form (t - µ)⊤ ∑-1(t - µ), and the MLE is obtained by minimizing this quadratic form. If the variance matrix is diagonal, the MLE becomes what is called a minimum weighted square error (MWSE) estimate. If all the diagonal terms of ∑ are equal, the MWSE becomes the minimum mean square error (MMSE) estimate. Thus, the MMSE estimator is optimal only under these assumptions cited above.

In this and following chapters, we will discuss random processes. After a brief introduction to this subject in Section 12.1, we will give an overview of various random processes in Section 12.2 and then discuss (strictly) stationary and wide-sense stationary random processes and introduce the notion of ergodicity. The last section focuses on complex-valued Gaussian processes, which will be useful in the study of communication systems and other applications.

Random process

There are many situations in which the time dependency of a set of probability functions is important. One example is a noise process that accompanies a signal process and should be suppressed or filtered out so that we can recover the signal reliably and accurately. Another example is the amount of outstanding packets yet to be processed at a network router or switch, which may lead to undesirable packet loss due to buffer overflows if not properly attended to in time.

Such a process can be conveniently characterized probabilistically by extending the notion of a random variable (RV) as follows: we assign to each sample point ω ∈ Ω a real-valued functionX(ω, t), where t is the time parameter or index parameter in some range T, which may be, for instance, T = (-∞, ∞) or T = {0, 1, 2, …} (see Figure 12.1). Imagine that we can observe this set of time functions {X(ω, t); ω ∈ Ω, t ∈ T} at some instant t = t1.

In Sections 4.2.4 and 4.3.1 we defined the normal (or Gaussian) distributions for both single and multiple variables and discussed their properties. The normal distribution plays a central role in the mathematical theory of statistics for at least two reasons. First, the normal distribution often describes a variety of physical quantities observed in the real world. In a communication system, for example, a received waveform is often a superposition of a desired signal waveform and (unwanted) noise process, and the amplitude of the noise is often normally distributed, because the source of such noise is usually what is known as thermal noise at the receiver front. The normality of thermal noise is a good example of manifestation in the real world of the CLT, which says that the sum of a large number of independent RVs, properly scaled, tends to be normally distributed. In Chapter 3 we saw that the binomial distribution and the Poisson distribution also tend to a normal distribution in the limit. We also discussed the CLT and asymptotic normality.

The second reason for the frequent use of the normal distribution is its mathematical tractability. For instance, sums of independent normal RVs are themselves normally distributed. Such reproductivity of the distribution is enjoyed only by a limited class of distributions (that is, binomial, gamma, Poisson). Many important results in the theory of statistics are founded on the assumption of a normal distribution.

Why study probability, random processes, and statistical analysis?

Many problems we face in daily life involve some degree of uncertainty and we need to use probabilistic reasoning in order to make a sound decision, be it an investment, medical, or social problem. In many cases “probability” may represent our personal judgment about how likely a particular event (e.g., price movement of a stock we own; positive or negative effect of some medicine we may choose to take when we are ill) is to occur. Probability attached to a given event is generally not based on any precise computation but is often a reasonable assessment based on our knowledge or experience. Such probability may be aptly called subjective or qualitative probability, and may not be scientifically estimated, unless the same event happens repeatedly.

The other type of probability we deal with is what we may call objective or quantitative probability, which can be estimated objectively based on empirical evidence from observable events. This philosophical question concerning subjective versus objective probability has been pondered by many probabilists and statisticians, as we will briefly discuss in Section 1.2. Philosophical discussion on subjective probability still continues today as a fascinating topic, as is found in arguments between two schools of statistics, i.e., frequentist statistics and Bayesian statistics, which will be discussed at the end of this chapter.

In this book we will discuss probability based on the modern probability theory established by Kolmogorov in the early twentieth century.