Optimization problems often have symmetries. For example, the value of the cost function may not change if its input vectors are scaled, translated or rotated. Then, it makes sense to quotient out the symmetries. If the quotient space is a manifold, it is called a quotient manifold. This often happens when the symmetries result from invariance to group actions: This chapter first reviews conditions for this to happen. Continuing with general quotient manifolds, the chapter reviews geometric concepts (points, tangent vectors, vector fields, retractions, Riemannian metrics, gradients, connections, Hessians and acceleration) to show how to work numerically with these abstract objects through lifts. The chapter aims to show the reader what it means to optimize on a quotient manifold, and how to do so on a computer. To this end, two important sections detail the relation between running Riemannian gradient descent and Newton’s method on the quotient manifold compared to running them on the non-quotiented manifold (called the total space). The running example is the Grassmann manifold as a quotient of the Stiefel manifold. Its tools are summarized in a closing section.

This chapter provides the classical definitions for manifolds (via charts, which have not appeared thus far), smooth maps to and from manifolds, tangent vectors and tangent spaces and differentials of smooth maps. Special care is taken to introduce the atlas topology on manifolds and to justify topological restrictions in the definition of a manifold. It is shown explicitly that embedded submanifolds of linear spaces as detailed in earlier chapters are manifolds in the general sense. Then, sections go on to explain how the geometric concepts introduced for embedded submanifolds extend to general manifolds mostly without effort. This includes tangent bundles, vector fields, retractions, local frames, Riemannian metrics, gradients, connections, Hessians, velocity and acceleration, geodesics and Taylor expansions. One section explains why the Lie bracket of two vector fields can be interpreted as a vector field (which we omitted in Chapter 5). The chapter closes with a section about submanifolds embedded in general manifolds (rather than only in linear spaces): This is useful in preparation for the next chapter.

To design more sophisticated optimization algorithms, we need more refined geometric tools. In particular, to define the Hessian of a cost function, we need a means to differentiate the gradient vector field. This chapter highlights why this requires care, then proceeds to define connections: the proper concept from differential geometry for this task. The proposed definition is stated somewhat differently from the usual: An optional section details why they are equivalent. Riemannian manifolds have a privileged connection called the Riemannian connection, which is used to define Riemannian Hessians. The same concept is used to differentiate vector fields along curves. Applied to the velocity vector field of a curve, this yields the notion of intrinsic acceleration; geodesics are the curves with zero intrinsic acceleration. The tools built in this chapter naturally lead to second-order Taylor expansions of cost functions along curves. These then motivate the definition of second-order retractions. Two optional closing sections further consider the important special case of Hessians on Riemannian submanifolds, and an intuitive way to build second-order retractions by projection.