Part IV of the book explains what is inside the code and how it works. This knowledge will make it possible to use the optimization code with more confidence, and more reliably. It will also help you know what to do when things do not work. The material presented will also be great preparation for further studies in optimization. With this more advanced knowledge, it will be possible to understand such things as the code error messages. The material examined in this part of the book will reinforce and advance the knowledge of optimization that was previously learned.

Metaphorically, the material up to this point taught us how to drive the design from a bad state to an optimal state. We did so without understanding how the car we are driving works, how to fix it, design it, or actually build it. However, we have sufficient practical knowledge to drive from a bad to a good design. The following material will teach us how the car works, how to fix it, how to make it perform very well, and even prepare us venture into the advanced world of designing cars (if that is our interest). That is, building optimization codes and algorithms. Interestingly, most of us want to use the power of optimization, without having to become expert in the intricate details of the inner-workings of the optimization code. This book provides the breadth of presentation to suit these diverse objectives. More advanced topics will be presented beyond Part IV.

1.1 Overview

Optimization can be viewed as a process that searches methodically for better answers, better solutions, or better designs that a human being may not be able to find through experience, intuition, or courageous trial-and-error. Optimization can be defined as the art of making things better. Interestingly, optimization very often does not simply allow us to do something better, but it may also make it possible to do something that we did not otherwise know how to do. To take full advantage of the power of optimization in practice, there is no choice but to use a computer.

The study of optimization typically takes a theoretical and/or computational approach. The theoretical approach is highly useful when the objective is to develop new optimization methods or to assess how the current methods work [1]. Additionally, the study of optimization often focuses on the understanding of various search algorithms for optimization. A book by Reklaitis and co-authors [2] is an example of methods-based optimization presentation. While these books play an important role in the general study of optimization, we pursue a different approach. The objective of this book takes on a practical perspective. The current interest is in the immediate ability to apply optimization in practice. In order to reach this objective, and to have the ability to apply optimization to real world problems, we use the approach that is almost always required in the application of optimization. We use the power of the computer, in conjunction with the study of different important practical aspects of optimization.

In order readily apply optimization in practice, the focus will be on the computational application of what we will learn, as we learn it. To do so, we use a computational modeling and coding tool that is powerful, easy to use, and one that is widely applied in engineering and other fields. Matlab is an excellent tool choice and will be used in this book. Matlab has arguably become the most popular tool for computational modeling worldwide.

Overview

This chapter covers the practice of optimization in diverse technical areas. What has been learned thus far is the practical knowledge that is needed to apply optimization to different types of designs or systems. We will explore optimizing systems/designs in the following disciplines: chemistry, mechanics, aerospace, automotive, mathematics (data fitting), nuclear, electrical, portfolio management, and business (Ref. [1]). Our ability to optimize these various designs provides us with the skills to apply optimization beyond the confines of this book or of the class you might be taking, and successfully venture into the real world!

Mechanical Engineering Example

Structural Example

Figure 10.1 represents a ten-bar truss. The members are connected to each other at six nodes, numbered 1 to 6. The truss is assumed to be planar, and every node has two degrees of freedom (i.e., it is free to move both in the x and y directions). The left edge of the truss is fixed to the wall. The displacements u1 to u8 are specified at the specific non-fixed nodes. F1 to F8 represent loads applied to these nodes. The variables by x1 to x10 denote the cross sectional areas of the truss members. The Young's modulus of the material is E = 1 × 106N/mm2. The maximum allowable stress in each bar is σult = ±100N/mm2 (i.e., tension/compression), and the maximum allowable deflection is δmax = ±2 mm. We note that the unit of length in the figure and in this problem is mm, and that of force is N. While the current example is commonly used in the literature, the reader is also encouraged to review the topology optimization survey by Rozvany in Ref. [2].

Imagine you are hired as the optimization expert for the ten-bar truss project. The structural engineer working on the project provides you with a program blackbox−10bar.m, which allows you to enter the loads at the nodes and the area of each member, and obtain the corresponding nodal displacements and stresses in the members. Figure 10.2 illustrates this task. The ten-bar truss is subjected to 10 different loading conditions given in Table 10.1.

In Part I, we presented some prerequisite material for our study of optimization: and introduction to Matlab, and some elementary mathematics. In Part II, we learned what we needed to explore what optimization is all about, and the important role it can play in your life as and engineer, scientist, business person, or anyone dealing with improving things quantitatively (e.g., performance and cost). In Part III, we get into the thick of things! We learn what we need to know to credibly address optimization problems in practice.

Specifically, the topics presented, with the chapter numbers, are given below:

Overview

In this chapter, we introduce efficient mathematical approaches to model the behavior of complex systems in the context of optimizing the system. Specifically, complex systems often involve a large number of design parameters (to be tuned), and the optimization of these systems often demand large scale computational simulations or experiments to quantify the system behavior. The resulting high dimensionality of the design space, the prohibitive computation time (or expense), or the lack of mathematical models present important challenges to the quantitative optimization of these complex systems. This chapter introduces traditional and contemporary approaches, such as design variable linking and surrogate modeling, to address the modeling challenges encountered in solving complex optimization problems.

Section 15.2 discusses the generic challenges in complex optimization problems. The impact of problem dimension is addressed in Sec. 15.3, where design variable linking and design of experiments are introduced. Section 15.4 presents surrogate modeling, where the discussion includes: the process, the polynomial response surface methodology, the radial basis function method, the Kriging method, and the artificial neural network method. The chapter concludes with a summary provided in Section. 15.5.

Modeling Challenges in Complex Optimization Problems

Modeling is one of the primary activities in optimization. Leveraging computational tools and models, as opposed to purely depending on experiments, can be considered the way forward in the area of system optimization. The design of complex systems, such as aircrafts, cars, and smart-grid networks, are increasingly performed using simulation-based design and analysis tools such as Finite Element Analysis (FEA) and Computational Fluid Dynamics (CFD). The incorporation of these tools has dramatically transformed modern engineering design and optimization approaches.

These changes, however, are not free of challenges. For example, modern aerospace systems present significantly complex design requirements [1], where design objectives generally involve improving performance, reducing costs, and maximizing safety. Another example is the optimization of a vehicle structure to absorb crash energy, maintain adequate passenger space, and control crash deceleration pulse during collisions [2].

Overview

The previous chapters presented deterministic optimization methods. These do not implicitly take into account the inherent uncertainties typically present in the design process, in the system being designed, or in the models describing the system behavior. Uncertainties emanates from myriad sources. These include imperfect manufacturing processes and material properties, fluctuating loading conditions, over-simplified engineering models, or uncertain operating environment. All of these may have a direct impact on the system performance in its use or in the market place. To obtain a reliable and robust design (these terms are defined later), these uncertainties must be considered as part of the design process.

In the past, empirical safety factors were often used to guard against engineering design failure [1]. Safety factors resulted in overly conservative designs, increasing the probability that businesses may lose their competitive edge in terms of cost and performance. More recently, design optimization methods under uncertainty have gained broad recognition. These approaches explicitly consider uncertainties of various forms, and search for designs that are insensitive to variations or uncertainty to a significant extent.

This chapter introduces the concept of design optimization under uncertainty, and discusses the pertinent popular approaches available. Section 16.2 defines a generic example that is used throughout the book to facilitate the presentation of the material. The next Section (Sec. 16.3) defines five generic components/STEPS involved in design under uncertainty. This is followed by the presentation of these STEPS in the following five Sections: (1) uncertainty types identification (Sec. 16.4), (2) uncertainty quantification (Sec. 16.5), (3) uncertainty propagation (Sec. 16.6), (4) formulation of optimization under uncertainty (Sec. 16.7), and (5) results analysis (Sec. 16.8). Section 16.9 briefly discussed other popular methods. A chapter summary is provided in Sec 16.10.

Chapter Example

This section defines a generic example that it used throughout the chapter to illustrate the various concepts involved in design under uncertainty. Consider the optimization of a two bar truss shown in Fig. 16.1.

This part introduces carefully chosen advanced topics that are appropriate for graduate students or undergraduate students who might pursue study in the design optimization field. These topics will generally be of great interest to industry practitioners as well. While these topics are advanced, a mathematically advanced presentation is not provided. Instead, an introductory presentation is provided, with an eye toward practical usefulness.

Specifically, the topics presented, with the chapter numbers, are given below:

Overview

Linear programming (LP) is a technique for optimizing a linear objective function, subject to linear equality and linear inequality constraints. Linear programming is an important field of optimization for several reasons (Refs. [1, 2]). Many practical problems in engineering and operations research can be expressed as linear programming problems. Linear programming problems can be solved in an easy, fast, and reliable manner.

Linear programming was first used in the field of economics, and still remains popular in the areas of management, economics, finance, and engineering. During World War II, George Dantzig of the U.S. Air Force used LP techniques for planning problems. He invented the Simplex method, which is one of the most popular methods for solving LP problems.

Linear programming is a well developed subject in operations research, and several references that focus on linear programming alone are available [3, 4, 5, 6]. The reader is encouraged to consult these references for a more detailed discussion of linear programming.

In this chapter, the basics of linear programming problems will be studied. First, an introduction of the basic terminology and important concepts in LP problems will be presented in Sec. 11.2. In Sec. 11.3, the graphical approach for solving LP problems will be discussed, and four types of possible solutions for an LP problem will be introduced. Details on the Matlab linear programming solver are presented in Sec. 11.4. Sections 11.5 and 11.6 discuss the various concepts involved in the Simplex method. In Sec. 11.7, the notion of duality and interior point methods are briefly discussed. Section 11.8 concludes the chapter with a summary.

Basics of Linear Programming

In this section, some basic terminology and definitions in linear programming will be discussed.

A generic linear programming problem consisting of linear equality and linear inequality constraints is given as where z is the linear objective function to be minimized;

Overview

Welcome to the fascinating world of optimization. Indeed, you will find optimization to be a powerful addition to your education or to your toolkit in the workplace. This brief initial chapter provides you with a clear perspective as to how the book will play a key role in your understanding of optimization. As discussed in the Preface, this book takes a squarely practical perspective. As such, your learning will involve practicing optimization using many problems. By the end of your study of this book, you should expect to be able to work with others to optimize practically any design or system and improve its performance. If the design you would like to optimize is in your technical area of expertise, you might be able to do all of the work yourself. If the design or system you would like to optimize involves modeling issues with which you are not familiar, you will simply need to collaborate with someone (or a team) who is able to provide you with the computational performance models. In this latter case, as it often is in practice, you will provide your knowledge of optimization, and someone else may model the system. In other words, one person might do the structural analysis, another might do the financial analysis, and you might use your knowledge of this book to optimize the combined system to make it competitive in the market place.

What Is Optimization? What Is Its Relation to Analysis and Design?

What is optimization? Optimization can be defined in different ways. Some define it as “the art of making things the best.” Interestingly, many people do not like that definition as it may not be reasonable, or even possible, to do something in the very best possible way. In practice, doing something as well as possible within practical constraints is very desirable. Designing a product and doing all we can to increase profit as much as is practically possible is also very desirable.

Overview

In this chapter, you will be exposed to perhaps some of the most important issues in optimization. Interestingly, most optimization books and courses leave it up to the students to learn many of these issues simply by chance, leaving a broad set of the important practical topics unaddressed. We are referring here to numerical conditioning issues which strongly control the success of computational optimization.

We will not engage in highly mathematical and theoretical issues of numerical optimization. Instead, we will learn about the tangible and practical issues that directly affect our success in formulating and solving optimization problems. Fortunately, for most of this chapter, all we need is the knowledge of first-year college mathematics.

Keeping these issues in mind will often make the difference between success and failure. That is: (i) easily applying optimization successfully, or (ii) experiencing great frustration in trying to obtain an adequate solution unsuccessfully. In addition, we will learn how to control the resulting accuracy of our results, a critical issue in practice. The topics explored here include: numerical conditioning, scaling, finite differences, automatic differentiation, termination criteria, and sensitivities of optimal solutions, as well as examples that illustrate how to handle these issues in practice (Refs. [1, 2, 3].

Numerical Conditioning—Algorithms, Matrices and Optimization Problems

We begin by asking the following questions: What is numerical conditioning? And how does it relate to optimization? As we have learned, optimization depends on the numerical evaluation of the performance of the system being optimized. This performance evaluation typically involves coding, simulation, or software-based mathematical analysis, often also involving matrix manipulations (e.g., involving Hessians). For example, linear programming involves extensive matrix manipulations. Therefore, understanding the numerical properties of the matrices and of the algorithms used in optimization codes is important. From a practical point of view, we can think of a numerically well-conditioned problem or matrix as one that lends itself to easy numerical computation. Conversely, we can think of a numerically ill-conditioned problem or matrix as one that lends itself to difficult numerical computation.

Overview

In this chapter, we introduce the important activities of analysis, design, optimization, and modeling. While we are all generally familiar with these terms, it is important to understand them in the context of how they relate to our optimization activities. We also develop an important understanding of how modeling system behavior is a distinct activity from modeling an optimization problem. These involve two distinct lines of expertise. In most of your courses (e.g., structures, dynamics and finance) you focused on the former. In this book, we focus on the latter.

Analysis, Design and Optimization

Analysis and optimization are two activities integral to the process of design (Refs. [1, 2, 3, 4, 5]). In this chapter, as well as in the remainder of the book, we will primarily focus on these two activities in the context of engineering or systems design. However, the mathematical concepts, approaches, algorithms, and software tools that you will learn in this book could be applicable to diverse fields beyond engineering (e.g., optimization of market portfolios) [6]. Although, from a technical perspective, analysis and optimization can be considered to be steps within the process of design, they can also be performed as stand-alone activities toward other end goals within the scope of academic research and industrial R&D. In this section, you will learn the definition of these activities and how they are related to each other. At the same time, you will have the opportunity to understand and appreciate the roles and responsibilities of the users, researchers, or engineers who execute these activities, whether individually or as a part of a team.

In order to help you better understand the practical essence of these activities, the following simple design example will be used throughout this section. You are designing a table that can carry as much weight (of objects placed on it) as possible, while fitting within a particular corner of your room. You are allowed a limited budget to construct the table.

Overview

In most previous chapters, continuous optimization problems were considered where the design variables were assumed to be continuous; that is, design variables assume real values within given ranges. In many practical engineering problems, the acceptable values of the design variables do not form a continuous set. These problems are referred to as discrete optimization problems. For example, the number of rivets required in a riveted joint has to be an integer (such as 1, 2, 3). Another example is when the feasible region of the design variable is a set of given discrete numbers, such as {6.25, 6.95, 7.65}, which may be the available standardized sizes of nuts. The basics of discrete optimization were introduced in Chapter 9, where some pertinent elementary methods were presented. This chapter introduces more advanced approaches. The reader is advised to first review Chapter 9 as preparation for the current chapter.

This chapter is organized as follows. The next section (Sec. 14.2) provides the problem classes, examples, and definition (along with the notion of computational complexity of the solution algorithms). Section 14.3 discusses the basics of some popular techniques used to solve integer programming problems, with examples. The methods studied will be: the exhaustive search method (Sec. 14.3.1), the graphical method (Sec. 14.3.2), the relaxation method (Sec. 14.3.3), the branch and bound method (Sec. 14.3.4), and the cutting plane method (Sec. 14.3.5). Popular current software options (Sec. 14.3.7) are also discussed. The chapter concludes with a summary in Sec. 14.4.

Problem Classes, Examples and Definition

This section presents discrete optimization problem classes, problem examples, and problem definition. Computational complexity of the solution algorithms is also briefly addressed in connection with the discrete optimization problem definition.

Overview

This book has presented various algorithms and applications where the optimizer was primarily gradient-based (i.e., the search direction is governed by gradient and/or Hessian information). This chapter introduces an entirely different class of optimization algorithms called the evolutionary algorithms (EA). Evolutionary algorithms imitate natural selection processes to develop powerful computational algorithms to select optimal solutions. Genetic algorithms (GA), simulated annealing (SA), ant colony optimization (ACO), particle swarm optimization (PSO), and tabu search (TS) are some of the popular techniques that fall under the umbrella of evolutionary algorithms.

The motivation for using biologically-inspired computational approaches stems from two key observation. First, the mathematical optimization algorithms in solving complex problems in engineering, computing, and other fields suffer strong limitations. The common challenges in these areas revolve around the lack of mathematical models that define the physical phenomena, discontinuous functions, and high nonlinearity. Second, many complex problems encountered in engineering already exist in nature in some relevant form. Optimization is inherent in nature, such as in the process of adaptation performed by biological organisms in order to survive. Engineers and scientists continue to explore the various efficient problem-solving techniques employed by nature to optimize natural systems.

The relative advantages and limitations of EAs vs. traditional optimization methods are as follows: