This appendix is a collection of topics that were slightly peripheral to the main flow of the book, but still potentially interesting to some readers. The derivations of Fisher's information matrix in several forms as well as Stein's lemma are both important tools employed in the main parts of the book.

With both our estimation and Lie group tools from previous chapters, we now begin to bring the two together. We discuss a classic three-dimensional estimation problem in robotics: pointcloud alignment; this gives us our first example of carrying out optimization over the group of rotations by a few different means. We then present the classic problem of localizing a moving robot using point observations of known three-dimensional landmarks; this involves adapting the extended Kalman filter (EKF) to work with the group of poses. Another common problem in robotics is that of pose-graph optimization, which is easily handled using our Lie group tools. We conclude with a presentation of how to carry out trajectory estimation based on an inertial measurement unit (IMU) both recursively via the EKF and batch using IMU preintegration for efficiency.

This chapter is devoted to the classic simultaneous localization and mapping (SLAM) problem and the related problem, bundle adjustment. In these problems we must estimate not only the trajectory of a robot but also the three-dimensional positions of many point landmarks, based on noisy sensor data and a motion model (in the case of SLAM). We discuss how to adapt the tools presented earlier to include landmarks in the state; the inclusion of landmarks changes the sparsity pattern of the resulting estimation equations and we discuss strategies of continuing to solve them efficiently. Our approach is carried out entirely in three dimensions using our Lie group tools.

As the book attempts to be as stand-alone as possible, this chapter provides up front a summary of all the results in probability theory that will be needed later on. Probability is key to estimation as we not only want to estimate, for example, where something is but how confident we are in that estimate. The first half of the chapter introduces general probability density functions, Bayes' theorem, the notion of independence, and quantifying uncertainty amongst other topics. The second half of the chapter delves into Gaussian probability density functions specifically and establishes the key tools needed in common estimation algorithms to follow in later chapters. This chapter can also simply serve as a reference for readers already familiar with the content.

This appendix serves as a quick summary of the main linear algebra and matrix calculus tools used throughout the book. It was designed primarily as a reference but could be used as a primer or refresher to be read before the main chapters of the book.

This chapter takes a step back and revisits nonlinear estimation through the lens of variational inference, another concept common in the machine learning world. Estimation is posed as minimizing a data-likelihood objective, the Kullback-Leibler divergence between a Gaussian estimate and the true Bayesian posterior. We follow through the consequences of this starting point and show that we can arrive at many of the algorithms presented earlier through appropriate approximations, but can also open the door to new possibilities. For example, a derivative-free batch estimator that uses sigmapoints is discussed. Variational inference also provides a principled approach to learning parameters in our estimators from training data (i.e., parameters of our motion and observation models).

This chapter presents an overview of VR systems from hardware (Section 2.1) to software (Section 2.2), including the introduction of the Virtual World Generator (VWG), which maintains the geometry and physics of the virtual world, to human perception (Section 2.3). The purpose is to quickly provide a sweeping perspective so that the detailed subjects in the remaining chapters will be understood within the larger context.

This chapter transitions from the physiology of human vision to perception. How do our brains interpret the world around us so effectively in spite of our limited biological hardware? To understand how we may be fooled by visual stimuli presented by a display, you must first understand how we perceive or interpret the real world under normal circumstances. It is not always clear what we will perceive. We have already seen several optical illusions. VR itself can be considered as a grand optical illusion. Under what conditions will it succeed or fail? Section 6.1 covers perception of the distance of objects from our eyes, which is also related to the perception of object scale. Section 6.2 explains how we perceive motion. An important part of this is the illusion of motion that we perceive from videos, which are merely a sequence of pictures. Section 6.3 covers the perception of color, which may help explain why displays use only three colors (red, green, and blue) to simulate the entire spectral power distribution of light. Finally, Section 6.4 presents a statistically based model of how information is combined from multiple sources to produce a perceptual experience.

This chapter introduces interaction mechanisms that may not have a counterpart in the physical world. Section 10.1 introduces general motor learning and control concepts. The most important concept is remapping, in which a motion in the real world may be mapped into a substantially different motion in the virtual world. This enables many powerful interaction mechanisms. The task is to develop ones that are easy to learn, easy to use, effective for the task, and provide a comfortable user experience. Section 10.2 discusses how the user may move himself in the virtual world, while remaining fixed in the real world. Section 10.3 presents ways in which the user may interact with other objects in the virtual world. Section 10.4 discusses social interaction mechanisms, which allow users to interact directly with each other. Section 10.5 briefly considers some additional interaction mechanisms, such as editing text, designing 3D structures, and Web browsing.

In the real world, audio is crucial to art, entertainment, and oral communication. Audio recording and reproduction can be considered a VR experience by itself, with both a CAVE-like version (surround sound) and a headset version (wearing headphones). When combined consistently with the visual component, audio helps provide a compelling and comfortable VR experience. Each section of this chapter is the auditory (or audio) complement to one of Chapters 4 through 7. The progression again goes from physics to physiology, and then from perception to rendering. Section 11.1 explains the physics of sound in terms of waves, propagation, and frequency analysis. Section 11.2 describes the parts of the human ear and their function. This naturally leads to auditory perception, which is the subject of Section 11.3. Section 11.4 concludes by presenting auditory rendering, which can produce sounds synthetically from models or reproduce captured sounds.

We now want to model motions more accurately because the physics of both real and virtual worlds impact VR experiences. The accelerations and velocities of moving bodies impact simulations in the VWG and tracking methods used to capture user motions in the physical world. Section 8.1 introduces fundamental concepts from math and physics, including velocities, accelerations, and the movement of rigid bodies. Section 8.2 presents the physiology and perceptual issues from the human vestibular system, which senses velocities and accelerations. Section 8.3 then describes how motions are described and produced in a VWG. This includes numerical integration and collision detection. Section 8.4 focuses on vection, which is a source of VR sickness that arises due to sensory conflict between the visual and vestibular systems: the eyes may perceive motion while the vestibular system is not fooled. This can be considered as competition between the physics of the real and virtual worlds.

This chapter covers the geometry part of the Virtual World Generator (VWG), which is needed to make models and move them around. The models could include the walls of a building, furniture, clouds in the sky, the user’s avatar, and so on. Section 3.1 covers the basics of how to define consistent, useful models. Section 3.2 explains how to apply mathematical transforms that move them around in the virtual world. This involves two components: translation (changing position) and rotation (changing orientation). Section 3.3 presents the best ways to express and manipulate 3D rotations, which are the most complicated part of moving models. Section 3.4 then covers how the virtual world appears if we try to “look” at it from a particular perspective. This is the geometric component of visual rendering. Finally, Section 3.5 puts all of the transformations together so that you can see how to go from defining a model to having it appear in the right place on the display.

This chapter considers what VR means in a way that captures the most crucial aspects in spite of rapidly changing technology. Relevant terminology is introduced. The subsequent discussion covers what VR is considered to be today and what we envision for its future. The chapter starts with two thought-provoking examples: (1) A human having an experience of flying over virtual San Francisco by flapping his own wings and (2) a gerbil running on a freely rotating ball while exploring a virtual maze that appears on a projection screen around the mouse.

This chapter addresses visual rendering, which specifies what the visual display should show through an interface to the virtual world generator (VWG). Sections 7.1 and 7.2 cover basic concepts at the core of computer graphics, and VR-specific issues. They mainly address the case of rendering for virtual worlds that are formed synthetically. Section 7.1 explains how to determine the light that should appear at a pixel based on light sources and the reflectance properties of materials that exist purely in the virtual world. Section 7.2 explains rasterization methods, which efficiently solve the rendering problem and are widely used in specialized graphics hardware, called GPUs. Section 7.3 addresses VR-specific problems that arise from imperfections in the optical system. Section 7.4 focuses on latency reduction, which is critical to VR, so that virtual objects appear in the right place at the right time. Otherwise, side effects could arise, such as VR sickness, fatigue, adaptation to flaws, or an unconvincing experience. Section 7.5 explains rendering for captured rather than synthetic virtual worlds. This covers VR experiences that are formed from panoramic photos and videos.