Overview

In this chapter, you will be exposed to perhaps some of the most important issues in optimization. Interestingly, most optimization books and courses leave it up to the students to learn many of these issues simply by chance, leaving a broad set of the important practical topics unaddressed. We are referring here to numerical conditioning issues which strongly control the success of computational optimization.

We will not engage in highly mathematical and theoretical issues of numerical optimization. Instead, we will learn about the tangible and practical issues that directly affect our success in formulating and solving optimization problems. Fortunately, for most of this chapter, all we need is the knowledge of first-year college mathematics.

Keeping these issues in mind will often make the difference between success and failure. That is: (i) easily applying optimization successfully, or (ii) experiencing great frustration in trying to obtain an adequate solution unsuccessfully. In addition, we will learn how to control the resulting accuracy of our results, a critical issue in practice. The topics explored here include: numerical conditioning, scaling, finite differences, automatic differentiation, termination criteria, and sensitivities of optimal solutions, as well as examples that illustrate how to handle these issues in practice (Refs. [1, 2, 3].

Numerical Conditioning—Algorithms, Matrices and Optimization Problems

We begin by asking the following questions: What is numerical conditioning? And how does it relate to optimization? As we have learned, optimization depends on the numerical evaluation of the performance of the system being optimized. This performance evaluation typically involves coding, simulation, or software-based mathematical analysis, often also involving matrix manipulations (e.g., involving Hessians). For example, linear programming involves extensive matrix manipulations. Therefore, understanding the numerical properties of the matrices and of the algorithms used in optimization codes is important. From a practical point of view, we can think of a numerically well-conditioned problem or matrix as one that lends itself to easy numerical computation. Conversely, we can think of a numerically ill-conditioned problem or matrix as one that lends itself to difficult numerical computation.