This book is an introduction to numerical analysis: the solution of mathematical problems using numerical algorithms. Typically these algorithms are implemented with computers. To solve numerically a problem in science or engineering one is typically faced with four concerns:

1. How can the science/engineering problem be posed as a mathematical problem?

2. How can the mathematical problem be solved at all, using a computer?

3. How can it be solved accurately?

4. How can it be solved quickly?

The first concern comes from the science and engineering disciplines, and is outside the scope of this book. However, there are many practical examples and problems drawn from engineering, chemistry, and economics applications.

The focus of most introductory texts on numerical methods is, appropriately, concern #2, and that is also the main emphasis here. Accordingly, a number of different subjects are described that facilitate solution of a wide array of science and engineering problems.

Accuracy, concern #3, deals with numerical error and approximation error. There is a brief introductory chapter on error that presents the main ideas. Throughout the remainder of this book, algorithm choices and implementation details that affect accuracy are described. For the most part, where a claim of accuracy is made an example is given to illuminate the point, and to show how such claims can be tested.

The goal of optimization is to find a vector solution x that minimizes some scalar function f (x). We will assume that the function is at least C2 (i.e., is continuous, has continuous derivatives and has continuous second derivatives). The solution space may be subject to constraints, e.g., the positivity of some elements of x.

We have seen already one method, conjugate gradients (Section 4.1), which assumes a specific quadratic function f with a minimum that solves the linear equation Ax = b. We are now interested in more complicated functions, and for many such functions the solution is notoriously dificult. For example, the function may possess any number of local minima. In the neighborhood of a local minimum, quasi-Newton type methods will converge to the local minimum, which may be distant from the desired global minimum. None of the approaches described here can cope with that problem – the success of the methods relies on smoothness of the function f (x) and the quality of the starting guess x(0).

We will examine separately the case of nonlinear functions f(x) without constraints (variable metric methods), and linear constraints applied to linear functions (linear programming). Methods for constrained nonlinear problems (nonlinear programming) are built on these simpler methods.

The variable metric methods for n-dimensional problems rely on accurate and stable methods for simpler one-dimensional (1D) problems. The problem of finding a local minimum in 1D is broken down into two parts. Problem I is called bracketing.