Tractability is the study of computational tasks with the goal of identifying which problem classes are tractable or, in other words, efficiently solvable. The class of tractable problems is traditionally assumed to be solvable in polynomial time by a deterministic Turing machine and is denoted by P.The class contains many natural tasks such as sorting a set of numbers, linear programming (the decision version), determining if a number is prime, and finding a maximum weight matching. Many interesting problems, however, lie in another class that generalizes P and is known as NP: the class of languages decidable in polynomial time on a non-deterministic Turing machine. We trivially have that P isasubset of NP (many researchers also believe that it is a strict subset). It is believed that many problems in the class NP are, in the worst case, intractable and do not admit efficient inference. Problems such as maximum stable set, the traveling salesman problem and graph coloring are known to be NP-hard (at least as hard as the hardest problems in NP). It is, therefore, widely suspected that there are no polynomial-time algorithms for NP-hard problems.

Many NP-hard problems in different areas such as AI [42], Database Systems [6, 81], Game theory [45, 31, 20], and Network Design [34], are known to be efficiently solvable when restricted to instances whose underlying structures can be modeled via acyclic graphs or acyclic hypergraphs. For such restricted classes of instances, solutions can usually be computed via dynamic programming. However, as a matter of fact, (graphical) structures arising from real applications are in most relevant cases not properly acyclic. Yet, they are often not very intricate and exhibit some rather limited degree of cyclicity, which suffices to retain most of the nice properties of acyclic instances.

Introduction

Machine Learning (ML) broadly encompasses a variety of adaptive, autonomous, and intelligent tasks where one must “learn” to predict from observations and feedback. Throughout its evolution, ML has drawn heavily and successfully on optimization algorithms; this relation to optimization is not surprising as “learning” and “adapting” ultimately involve problems where some quality function must be optimized.

But the interaction between ML and optimization is now undergoing rapid change. The increased size, complexity, and variety of ML problems, not only prompts a refinement of existing optimization techniques, but also spurs development of new methods tuned to the specific needs of ML applications.

In particular, ML applications must usually cope with large-scale data, which forces us to prefer “simpler,” perhaps less accurate but more scalable algorithms. Such methods can also crunch through more data, and may actually be better suited for learning – for a more precise characterization see [11]. The use of possibly less accurate methods is also grounded in pragmatic concerns: modeling limitations, observational noise, uncertainty, and computational errors are pervasive in real data.

Introduction

NP-hard optimization problems are ubiquitous, and unless P=NP, we cannot expect algorithms that find optimal solutions on all instances in polynomial time. This intractability thus forces us to relax one of the three above mentioned constraints. Approximation algorithms relax the optimality constraint, and aim to do so by as small an amount as possible. We shall concern ourselves with discrete optimization problems, where the goal is to find amongst the set of feasible solutions, the one that minimizes (or maximizes) the value of the objective function. Usually, the space of feasible solutions is defined implicitly, e.g. the set of cuts in a graph on n vertices. The objective function associates with each feasible solution a real value; this usually has a succinct representation as well, e.g. the number of edges in the cut. We measure the performance of an approximation algorithm on a given instance by the ratio of the value of the solution output by the algorithm, to that of the optimal solution.

Submodularity is a property of set functions with deep theoretical consequences and far-reaching applications. At first glance it seems very similar to concavity, in other ways it resembles convexity. It appears in a wide variety of applications: in Computer Science it has recently been identified and utilized in domains such as viral marketing [39], information gathering [44], image segmentation [10, 40, 36], document summarization [56], and speeding up satisfiability solvers [73]. Our emphasis in this chapter is on maximization; there are many important results and applications related to minimizing submodular functions that we do not cover.

As a concrete running example, we will consider the problem of deploying sensors in a drinking water distribution network (see Figure 3.1) in order to detect contamination. In this domain, we may have a model of how contaminants, accidentally or maliciously introduced into the network, spread over time. Such a model then allows to quantify the benefit f(A) of deploying sensors at a particular set A of locations (junctions or pipes in the network) in terms of the detection performance (such as average time to detection).

Based on this notion of utility, we then wish to find an optimal subset A ⊆ V of locations maximizing the utility, maxAf(A), subject to some constraints (such as bounded cost). This application requires solving a difficult real-world optimization problem, that can be handled with the techniques discussed in this chapter (Krause et al. [49] show in detail how submodular optimization can be applied in this domain.)

Chapter Overview

Every engineered communication system, ranging from satellite communications to hard disk drives to Ethernet must operate under noisy conditions. The key to reliable communication and storage is to add an appropriate amount of redundancy to make the system reliable. The field of channel coding is concerned with constructing channel codes and their decoding algorithms: controlled redundancy is introduced into a message prior to its transmission over a noisy channel (the encoding step), and this redundancy is removed from the received noisy string to unveil the intended message (the decoding step). The encoded message is referred to as the codeword. The collection of all codewords is a channel code. Assuming all the messages have the same length, and all the codewords have the same length, the ratio of message length to codeword length is the code rate. To make coding systems implementable in practice, channel codes must provide the best possible protection to noise while their decoding algorithms must be of acceptable complexity.

There is a clear tension with this dual goal: if a channel code protects a fairly long encoded message with relatively few but carefully derived redundancy bits (necessary for high performance), the optimal, maximum likelihood decoding algorithm has exponential complexity.

Introduction

Boolean Satisfiability (SAT) is an NP-complete decision problem [14]. SAT was the first problem to be shown NP-complete. There are no known polynomial time algorithms for SAT. Moreover, it is believed that any algorithm that solves SAT is exponential in the number of variables, in the worst-case.

Although SAT is in theory an NP-complete problem, in practice it can be seen as a success story of Computer Science. There have been remarkable improvements since the mid 90s, namely clause learning and unique implication points (UIPs) [43], search restarts [15, 26], lazy data structures [48], adaptive branching heuristics [48], clause minimization [59] and preprocessing [18].

Introduction

Computational problems from many different areas involve finding values for variables that satisfy certain specified restrictions and optimise certain specified criteria.

In this chapter, we will show that it is useful to abstract the general form of such problems to obtain a single generic framework. Bringing all such problems into a common framework draws attention to common aspects that they all share, and allows very general analytical approaches to be developed. We will survey some of these approaches, and the results that have been obtained by using them.

The generic framework we shall use is the valued constraint satisfaction problem (VCSP), defined formally in Section 4.3. We will show that many combinatorial optimisation problems can be conveniently expressed in this framework, and we will focus on finding restrictions to the general problem which are sufficient to ensure tractability.

An important and well-studied special case of the VCSP is the constraint satisfaction problem (CSP), which deals with combinatorial search problems which have no optimisation criteria. We give a brief introduction to the CSP in Section 4.2, before defining the more general VCSP framework in Section 4.3. Section 4.4 then presents a number of examples of problems that can be seen as special cases of the VCSP.

The remainder of the chapter discusses what happens to the complexity of the valued constraint satisfaction problem when we restrict it in various ways.