Semidefinite programming (SDP) is an optimization model with vector or matrix variables, where the objective to be minimized is linear, and the constraints involve affine combinations of symmetric matrices that are required to be positive (or negative) semidefinite. SDPs include as special cases LPs, QCQPs, and SOCPs; they are perhaps the most powerful class of convex optimization models with specific structure, for which efficient and well-developed numerical solution algorithms are currently available.

SDPs arise in a wide range of applications. For example, they can be used as sophisticated relaxations (approximations) of non-convex problems, such as Boolean problems with quadratic objective, or rank-constrained problems. They are useful in the context of stability analysis or, more generally, in control design for linear dynamical systems. They are also used, to mention just a few, in geometric problems, in system identification, in algebraic geometry, and in matrix completion problems under sparsity constraints.

11.1 From linear to conic models

In the late 1980s, researchers were trying to generalize linear programming. At that time, LP was known to be solvable efficiently, in time roughly cubic in the number of variables or constraints. The new interior-point methods for LP had just become available, and their excellent practical performance matched the theoretical complexity bounds. It seemed, however, that, beyond linear problems, one encountered a wall.

Optimization refers to a branch of applied mathematics concerned with the minimization or maximization of a certain function, possibly under constraints. The birth of the field can perhaps be traced back to an astronomy problem solved by the young Gauss. It matured later with advances in physics, notably mechanics, where natural phenomena were described as the result of the minimization of certain “energy” functions. Optimization has evolved towards the study and application of algorithms to solve mathematical problems on computers.

Today, the field is at the intersection of many disciplines, ranging from statistics, to dynamical systems and control, complexity theory, and algorithms. It is applied to a widening array of contexts, including machine learning and information retrieval, engineering design, economics, finance, and management. With the advent of massive data sets, optimization is now viewed as a crucial component of the nascent field of data science.

In the last two decades, there has been a renewed interest in the field of optimization and its applications. One of the most exciting developments involves a special kind of optimization, convex optimization. Convex models provide a reliable, practical platform on which to build the development of reliable problem-solving software. With the help of user-friendly software packages, modelers can now quickly develop extremely efficient code to solve a very rich library of convex problems.

In this chapter we present a compact selection of numerical algorithms for performing basic matrix computations. Specifically, we describe the power iteration method for computing eigenvalues and eigenvectors of square matrices (along with some of its variants, and a version suitable for computing SVD factors); we discuss iterative algorithms for solving square systems of linear equations, and we detail the construction of the QR factorization for rectangular matrices.

7.1 Computing eigenvalues and eigenvectors

7.1.1 The power iteration method

In this section we outline a technique for computing eigenvalues and eigenvectors of a diagonalizable matrix. The power iteration (PI) method is perhaps the simplest technique for computing one eigenvalue/eigenvector pair for a matrix. It has rather slow convergence and it is subject to some limitations. However, we present it here since it forms the building block of many other more refined algorithms for eigenvalue computation, such as the Hessenberg QR algorithm, and also because interest in the PI method has been recently revived by applications to very large-scale matrices, such as the ones arising in web-related problems (e.g., Google PageRank). Many other techniques exist for computing eigenvalues and eigenvectors, some of them tailored for matrices with special structure, such as sparse, banded, or symmetric. Such algorithms are described in standard texts on numerical linear algebra.

Optimization is a technology that can be used to devise effective decisions or predictions in a variety of contexts, ranging from production planning to engineering design and finance, to mention just a few. In simplified terms, the process for reaching the decision starts with a phase of construction of a suitable mathematical model for a concrete problem, followed by a phase where the model is solved by means of suitable numerical algorithms. An optimization model typically requires the specification of a quantitative objective criterion of goodness for our decision, which we wish to maximize (or, alternatively, a criterion of cost, which we wish to minimize), as well as the specification of constraints, representing the physical limits of our decision actions, budgets on resources, design requirements that need be met, etc. An optimal design is one which gives the best possible objective value, while satisfying all problem constraints.

In this chapter, we provide an overview of the main concepts and building blocks of an optimization problem, along with a brief historical perspective of the field. Many concepts in this chapter are introduced without formal definition; more rigorous formalizations are provided in the subsequent chapters.

1.1 Motivating examples

We next describe a few simple but practical examples where optimization problems arise naturally. Many other more sophisticated examples and applications will be discussed throughout the book.

Dynamical systems are physical systems evolving in time. Typically, these systems are modeled mathematically by a system of ordinary differential equations, in which command and disturbance signals act as inputs that, together with pre-existing initial conditions, generate the evolution in time of internal variables (the states) and of output signals. These kinds of model are ubiquitous in engineering, and are used to describe, for instance, the behavior of an airplane, the functioning of a combustion engine, the dynamics of a robot manipulator, or the trajectory of a missile.

Broadly speaking, the problem of control of a dynamical system amounts to determining suitable input signals so as to make the system behave in a desired way, e.g., to follow a desired output trajectory, to be resilient to disturbances, etc. Even an elementary treatment of control of dynamical systems would require an entire text-book. Here, we simply focus on very few specific aspects related to a restricted class of dynamical systems, namely finite-dimensional, linear, and time-invariant systems.

We start our discussion by introducing continuous-time models and their discrete-time counterparts. For discrete-time models, we highlight the connections between the input-output behavior over a finite horizon, and static linear maps described by systems of linear equations. We shall show how certain optimization problems arise naturally in this context, and discuss their interpretation in a control setting.

A vector is a collection of numbers, arranged in a column or a row, which can be thought of as the coordinates of a point in n-dimensional space. Equipping vectors with sum and scalar multiplication allows us to define notions such as independence, span, subspaces, and dimension. Further, the scalar product introduces a notion of the angle between two vectors, and induces the concept of length, or norm. Via the scalar product, we can also view a vector as a linear function. We can compute the projection of a vector onto a line defined by another vector, onto a plane, or more generally onto a subspace. Projections can be viewed as a first elementary optimization problem (finding the point in a given set at minimum distance from a given point), and they constitute a basic ingredient in many processing and visualization techniques for high-dimensional data.

2.1 Vector basics

2.1.1 Vectors as collections of numbers

Vectors are a way to represent and manipulate a single collection of numbers. A vector x can thus be defined as a collection of elements x1, x2, …, xn, arranged in a column or in a row. We usually write vectors in column format:

Element xi is said to be the i-th component (or the i-th element, or entry) of vector x, and the number n of components is usually referred to as the dimension of x.