This chapter covers the geometry part of the Virtual World Generator (VWG), which is needed to make models and move them around. The models could include the walls of a building, furniture, clouds in the sky, the user’s avatar, and so on. Section 3.1 covers the basics of how to define consistent, useful models. Section 3.2 explains how to apply mathematical transforms that move them around in the virtual world. This involves two components: translation (changing position) and rotation (changing orientation). Section 3.3 presents the best ways to express and manipulate 3D rotations, which are the most complicated part of moving models. Section 3.4 then covers how the virtual world appears if we try to “look” at it from a particular perspective. This is the geometric component of visual rendering. Finally, Section 3.5 puts all of the transformations together so that you can see how to go from defining a model to having it appear in the right place on the display.

This chapter considers what VR means in a way that captures the most crucial aspects in spite of rapidly changing technology. Relevant terminology is introduced. The subsequent discussion covers what VR is considered to be today and what we envision for its future. The chapter starts with two thought-provoking examples: (1) A human having an experience of flying over virtual San Francisco by flapping his own wings and (2) a gerbil running on a freely rotating ball while exploring a virtual maze that appears on a projection screen around the mouse.

This chapter addresses visual rendering, which specifies what the visual display should show through an interface to the virtual world generator (VWG). Sections 7.1 and 7.2 cover basic concepts at the core of computer graphics, and VR-specific issues. They mainly address the case of rendering for virtual worlds that are formed synthetically. Section 7.1 explains how to determine the light that should appear at a pixel based on light sources and the reflectance properties of materials that exist purely in the virtual world. Section 7.2 explains rasterization methods, which efficiently solve the rendering problem and are widely used in specialized graphics hardware, called GPUs. Section 7.3 addresses VR-specific problems that arise from imperfections in the optical system. Section 7.4 focuses on latency reduction, which is critical to VR, so that virtual objects appear in the right place at the right time. Otherwise, side effects could arise, such as VR sickness, fatigue, adaptation to flaws, or an unconvincing experience. Section 7.5 explains rendering for captured rather than synthetic virtual worlds. This covers VR experiences that are formed from panoramic photos and videos.

We now want to model motions more accurately because the physics of both real and virtual worlds impact VR experiences. The accelerations and velocities of moving bodies impact simulations in the VWG and tracking methods used to capture user motions in the physical world. Section 8.1 introduces fundamental concepts from math and physics, including velocities, accelerations, and the movement of rigid bodies. Section 8.2 presents the physiology and perceptual issues from the human vestibular system, which senses velocities and accelerations. Section 8.3 then describes how motions are described and produced in a VWG. This includes numerical integration and collision detection. Section 8.4 focuses on vection, which is a source of VR sickness that arises due to sensory conflict between the visual and vestibular systems: the eyes may perceive motion while the vestibular system is not fooled. This can be considered as competition between the physics of the real and virtual worlds.

Opportunities for failure exist at all levels, from hardware, to low-level software, to content creation engines. As hardware and low-level software rapidly improve, the burden is shifting more to developers of software engines and VR experiences. This chapter presents several topics that may aid engineers and developers in their quest to build better VR systems and experiences. Section 12.1 introduces methods for guiding them to improve their discriminatory power. Rather than adapting to become oblivious to a problem, a developer could train herself to become more sensitive to problems. Section 12.2 applies the fundamentals from this book to provide simple advice for VR developers. Section 12.3 covers VR sickness, including the main symptoms and causes, so that VR systems and experiences may be improved. Section 12.4 introduces general methods for designing experiments that involve human subjects, and includes some specific methods from psychophysics. All of the concepts from this chapter should be used to gain critical feedback and avoid pitfalls in an iterative VR development process.

We will see in this chapter that the apparent perfection of our vision is mostly an illusion because neural structures are filling in plausible details to generate a coherent picture in our heads that is consistent with our life experiences. When building VR technology that co-opts these processes, it important to understand how they work. They were designed to do more with less, and fooling these processes with VR produces many unexpected side effects because the display technology is not a perfect replica of the surrounding world. Section 5.1 discusses the anatomy of the human eye within the optical system. Most of the section is about photoreceptors, which are the “input pixels“ that get paired with the “output pixels” of a digital display for VR. Section 5.2 offers a taste of neuroscience by explaining what is known about the visual information that hierarchically propagates from the photoreceptors up to the visual cortex. Section 5.3 explains how our eyes move, which incessantly interferes with the images in our retinas. Section 5.4 concludes the chapter by applying the knowledge gained about visual physiology to determine VR display requirements, such as the screen resolution.

Knowing how light propagates in the physical world is crucial to understanding VR. One reason is the interface between visual displays and our eyes. Light is emitted from displays and arrives on our retinas in a way that convincingly reproduces how light arrives through normal vision in the physical world. In the current generation of VR headsets, a system of both engineered and natural lenses (parts of our eyes) guides the light. Another reason to study light propagation is the construction of virtual worlds. Section 4.1 covers basic physical properties of light, including its interaction with materials and its spectral properties. Section 4.2 provides idealized models of how lenses work. Section 4.3 then shows many ways that lens behavior deviates from the ideal model, thereby degrading VR experiences. Section 4.4 introduces the human eye as an optical system of lenses. Cameras, which can be considered as engineered eyes, are introduced in Section 4.5. Finally, Section 4.6 briefly covers visual display technologies, which emit light that is intended for consumption by human eyes.

This chapter surveys some topics that could influence widespread VR usage in the future, but are currently in a research and development stage. Sections 13.1 and 13.2 cover the forgotten senses. Earlier in this book, we covered vision, hearing, and balance (vestibular) senses, which leaves touch, smell, and taste. Section 13.1 covers touch, or more generally, the somatosensory system. This includes physiology, perception, and engineering technology that stimulates the somatosensory system. Section 13.2 covers the two chemical senses, smell and taste, along with attempts to engineer “displays” for them. Section 13.3 discusses how robots are used for telepresence and how they may ultimately become our surrogate selves through which the real world can be explored with a VR interface. Just like there are avatars in a virtual world (Section 10.4), the robot becomes a kind of physical avatar in the real world. Finally, Section 13.4 discusses steps toward the ultimate level of human augmentation and interaction: brain–machine interfaces.