The probabilistic method is a powerful method to show the existence of objects (e.g., graphs, functions) with certain desirable properties. We have already seen it used in Chapter 6 to show the existence of functions with high-circuit complexity, in Chapter 19 to show the existence of good error-correcting codes, and in several other places in this book. But sometimes the mere existence of an object is not enough: We need an explicit and efficient construction. This chapter provides such constructions for two well-known (and related) families of pseudorandom objects, expanders and extractors. They are important in computer science because they can often be used to replace (or reduce) the amount of randomness needed in certain settings. This is reminiscent of derandomization, the topic of Chapter 20, and indeed we will see several connections to derandomization throughout the chapter. However, a big difference between Chapter 20 and this one is that all results proven here are unconditional, in other words do not rely on unproven assumptions. Another topic that is related to expanders is constructions of error-correcting codes and related hardness-amplification results, which we saw in Chapter 19. For a brief discussion of the many deep and fascinating connections between codes, expanders, pseudorandom generators, and extractors, see the chapter notes.

Expanders are graphs whose connectivity properties (how many edges lie between every two sets A, B of vertices) are similar to those of “random” graphs–in this sense they are “pseudorandom” or “like random.”

The Turing machine model captures computations on bits (equivalently, integers), but many natural and useful algorithms are most naturally described as operating on uncountable sets such as the real numbers ℝ or complex numbers ℂ. A simple example is Newton's method for finding roots of a given real-valued function f. It iteratively produces a sequence of candidate solutions x0, x1, x2, …, ∈ ℝ where xi+1 = xi−f (xi)/f′ (xi). Under appropriate conditions this sequence can be shown to converge to a root of f. Likewise, a wide variety of algorithms in numerical analysis, signal processing, computational geometry, robotics, and symbolic algebra typically assume that a basic computational step involves an operation (+, ×, /) in some arbitrary field F. Such algorithms are studied in a field called computer algebra [vzGG99].

One could defensibly argue that allowing arbitrary field operations in an algorithm is unrealistic (at least for fields such as ℝ) since real-life computers can only do arithmetic using finite precision. Indeed, in practice, algorithms like Newton's method have to be carefully implemented within the constraints imposed by finite precision arithmetic. In this chapter though, we take a different approach and study models which do allow arithmetic operations on real numbers (or numbers from fields other than ℝ). Such an idealized model may not be directly implementable but it provides a useful approximation to the asymptotic behavior as computers are allowed to use more and more precision in their computations.

Complexity theory studies the computational hardness of functions. In this chapter we are interested in functions that are hard to compute on the “average” instance, continuing a topic that played an important role in Chapters 9 and 18 and will do so again in Chapter 20. The special focus in this chapter is on techniques for amplifying hardness, which is useful in a host of contexts. In cryptography (see Chapter 9), hard functions are necessary to achieve secure encryption schemes of nontrivial key size. Many conjectured hard functions like factoring are only hard on a few instances, not all. Thus these functions do not suffice for some cryptographic applications, but via hardness amplification we can turn them into functions that do suffice. Another powerful application will be shown in Chapter 20–derandomization of the class BPP under worst-case complexity theoretic assumptions. Figure 19.1 contains a schematic view of this chapter's sections and the way their results are related to that result. In addition to their applications in complexity theory, the ideas covered in this chapter have had other uses, including new constructions of error-correcting codes and new algorithms in machine learning.

For simplicity we study hardness amplification in context of Boolean functions though this notion can apply to functions that are not Boolean-valued. Section 19.1 introduces the first technique for hardness amplification, namely, Yao's XOR Lemma.

The problem of mathematically modeling computation may at first seem insurmountable: Throughout history people have been solving computational tasks using a wide variety of methods, ranging from intuition and “eureka” moments to mechanical devices such as abacus or sliderules to modern computers. Besides that, other organisms and systems in nature are also faced with and solve computational tasks every day using a bewildering array of mechanisms. How can you find a simple mathematical model that captures all of these ways to compute? The problem is even further exacerbated since in this book we are interested in issues of computational efficiency. Here, at first glance, it seems that we have to be very careful about our choice of a computational model, since even a kid knows that whether or not a new video game program is “efficiently computable” depends upon his computer's hardware.

In defining NP we sought to capture the phenomenon whereby if certain statements (such as “this Boolean formula is satisfiable”) are true, then there is a short certificate to this effect. Furthermore, we introduced the conjecture NP ≠ coNP according to which certain types of statements (such as “this Boolean formula is not satisfiable”) do not have short certificates in general. In this chapter we are interested in investigating this phenomenon more carefully, especially in settings where the existence of a short certificate is not obvious.

We start in Section 15.1 with some motivating examples. In Section 15.2 we formalize the notion of a proof system using a very simple example, propositional proofs. We also prove exponential lower bounds for the resolution proof system using two methods that serve as simple examples of important techniques in proof complexity. Section 15.3 surveys some other proof systems that have been studied and lower bounds known for them. Finally, Section 15.4 presents some metamathematical ruminations about whether proof complexity can shed some light on the difficulty of resolving P versus NP. There is a related, equally interesting question of finding short certificates assuming they exist, which we will mostly ignore except in the chapter notes.

Computational complexity theory has developed rapidly in the past three decades. The list of surprising and fundamental results proved since 1990 alone could fill a book: These include new probabilistic definitions of classical complexity classes (IP = PSPACE and the PCP theorems) and their implications for the field of approximation algorithms, Shor's algorithm to factor integers using a quantum computer, an understanding of why current approaches to the famous P versus NP will not be successful, a theory of derandomization and pseudorandomness based upon computational hardness, and beautiful constructions of pseudorandom objects such as extractors and expanders.

This book aims to describe such recent achievements of complexity theory in the context of more classical results. It is intendedto serve both as a textbook and as a reference for self-study. This means it must simultaneously cater to many audiences, and it is carefully designed with that goal in mind. We assume essentially no computational background and very minimal mathematical background, which we review in Appendix A. We have also provided a Web site for this book at http://www.cs.princeton.edu/theory/complexity with related auxiliary material, including detailed teaching plans for courses based on this book, a draft of all the book's chapters, andlinks to other online resources covering related topics. Throughout the book we explain the context in which a certain notion is useful, and why things are defined in a certain way. We also illustrate key definitions with examples.

We have already encountered some ways of “capturing” the essence of families of computational problems by showing that they are complete for some natural complexity class. This chapter continues this process by studying another family of natural problems (including one mentioned in Shannon's quote at the begginning of this chapter) whose essence is not captured by nondeterminism alone. The correct complexity class that captures these problems is the polynomial hierarchy, denoted PH, which is a generalization of P, NP and coNP. It consists of an infinite number of subclasses (called levels) each of which is important in its own right. These subclasses are conjectured to be distinct, and this conjecture is a stronger form of P ≠ NP. This conjecture tends to crop up (sometimes unexpectedly) in many complexity theoretic investigations, including in Chapters 6, 7, and 17 of this book.

In this chapter we provide three equivalent definitions of the polynomial hierarchy:

1. In Section 5.2 we define the polynomial hierarchy as the set of languages defined via polynomial-time predicates combined with a constant number of alternating forall (∀) and exists (∃) quantifiers, generalizing the definitions of NP and coNP from Chapter 2.

2. […]

This chapter describes the PCP Theorem, a surprising discovery of complexity theory, with many implications to algorithm design. Since the discovery of NP-completeness in 1972 researchers had mulled over the issue of whether we can efficiently compute approximate solutions to NP-hard optimization problems. They failed to design such approximation algorithms for most problems (see Section 11.1 for an introduction to approximation algorithms). They then tried to show that computing approximate solutions is also hard, but apart from a few isolated successes this effort also stalled.

The notion of computation has existed in some form for thousands of years, in contexts as varied as routine account keeping and astronomy. Here are three examples of tasks that we may wish to solve using computation:

• Given two integer numbers, compute their product.

• Given a set of n linear equations over n variables, find a solution, if it exists.

• Given a list of acquaintances and a list of all pairs among them who do not get along, find the largest set of acquaintances you can invite to a dinner party such that every two invitees get along with one another.

Throughout history people had a notion of a process of producing an output from a set of inputs in a finite number of steps, and they thought of “computation” as “a person writing numbers on a scratch pad following certain rules.”