An important operation in signal processing and machine learning is dimensionality reduction. There are many such methods, but the starting point is usually linear methods that map data to a lower-dimensional set called a subspace. When working with matrices, the notion of dimension is quantified by rank. This chapter reviews subspaces, span, dimension, rank, and nullspace. These linear algebra concepts are crucial to thoroughly understanding the SVD, a primary tool for the rest of the book (and beyond). The chapter concludes with a machine learning application, signal classification by nearest subspace, that builds on all the concepts of the chapter.

Chapter 8: Many problems in applied mathematics involve finding a minimum-norm solution or a best approximation, subject to certain constraints. Orthogonal subspaces arise frequently in solving such problems. Among the topics we discuss in this chapter are the minimum-norm solution to a consistent linear system, a least-squares solution to an inconsistent linear system, and orthogonal projections.

We consider the existence of closest points in convex subsets of Hilbert sapces. In particular this enables us to define the orthogonal projection onto a closed linear subspace U of a Hilbert space H, and thereby decompose any element x of H as x=u+v, where u is an element of U and v is contained in its orthogonal complement. We also discuss ‘best approximations’ of elements of H in spaces spanned by collections of elements of H.