An important class of NP-complete problems is that of constraint satisfaction problems (CSPs), which have been widely investigated and where a phase transition has been found to occur (Williams and Hogg, 1994; Smith and Dyer, 1996; Prosser, 1996). Constraint satisfaction problems are the analogue of SAT problems in first-order logic; actually, any finite CSP instance can be transformed into a SAT problem in an automatic way, as will be described in Section 8.4.

Formally, a finite CSP is a triple (X, R, D). Here X = {xi|1 ≤ i ≤ n} is a set of variables and R = {Rh, 1 ≤ h ≤ m} is a set of relations, each defining a constraint on a subset of variables in X; D = {Di|1 ≤ i ≤ n} is a set of variable domains Di such that every variable xi takes values only in the Di, whose cardinality |Di| equals di. The constraint satisfaction problem consists in finding an assignment in Di for each variable xi ∈ X that satisfies all relations in R.

In principle a relation Rh may involve any proper or improper subset of X. Nevertheless, most authors restrict investigation to binary constraints, defined as relations over two variables only. This restriction does not affect the generality of the results that can be obtained because any relation of arity higher than two can always be transformed into an equivalent conjunction of binary relations.

Machine learning's roots

Learning involves vital functions at different levels of consciousness, starting with the recognition of sensory stimuli up to the acquisition of complex notions for sophisticated abstract reasoning. Even though learning escapes precise definition there is general agreement on Langley's idea (Langley, 1986) of learning as a set of “mechanisms through which intelligent agents improve their behavior over time”, which seems reasonable once a sufficiently broad view of “agent” is taken. Machine learning has its roots in several disciplines, notably statistics, pattern recognition, the cognitive sciences, and control theory. Its main goal is to help humans in constructing programs that cannot be built up manually and programs that learn from experience. Another goal of machine learning is to provide computational models for human learning, thus supporting cognitive studies of learning.

Classification

Among the large variety of tasks that constitute the body of machine learning, one has received attention from the beginning: the acquiring of knowledge for performing classification. From this perspective machine learning can be described roughly as the process of discovering regularities from a set of available data and extrapolating these regularities to new data.

Machine learning as an algorithm

Over the years, machine learning has been understood in different ways. At first it was considered mainly as an algorithmic process. One of the first approaches to automated learning was proposed by Gold in his “learning in the limit” paradigm (Gold, 1967).

In this chapter we discuss a theory for the freezing of an isotropic liquid into a crystalline solid state with long-range order. The transformation is a first-order phase transition with finite latent heat absorbed in the process. We focus on a first-principles orderparameter theory of freezing that originated from the pioneering work of Ramakrishnan and Yussouff (1979). The theory approaches the problem from the liquid side and views the crystal as a liquid with grossly inhomogeneous density characterized by a lower symmetry of the corresponding lattice. This is in contrast to description of the crystal in terms of phonons. The crucial quantity characterizing the physical state of the system in this non-phonon-based model is the average one-particle density function n(x). The thermodynamic description of either phase involves a corresponding extremum principle for a relevant potential. The latter, obtained as a functional of the one-particle density n(x) and the stable thermodynamic state of the system, is identified by the corresponding density required for invoking the extremum principle. This approach, which is generally referred to as the density-functional theory (DFT) of freezing (Haymet, 1987; Baus, 1987, 1990; Singh, 1991; Löwen, 1994; Ashcroft 1996), has been improved over the years and successfully applied for the study of liquid-to-crystal transitions in various simple liquids, the solid–liquid interface, two-dimensional systems, metastable glassy states, etc. For applications of density-functional methods in statistical mechanics there exist general reviews (Evans, 1979; Henderson, 1992).

In the previous chapter we showed how the covering test in relational learning exhibits a phase transition associated with a complexity peak, for control parameter values typical of the problems investigated by current relational learners. We also showed that the complexity associated with the phase transition in matching can be partially tamed using smart search algorithms. However, as soon as the number of variables increases a little (say, from four to five) the complexity is again a strongly limiting factor for learning, because a leamer must face hundreds of thousands of matching problems during its search for hypotheses (formulas).

Leaving aside the problems caused by the computational complexity of matching, one may wonder whether the presence of a phase transition has additional effects on learning, for instance whether it affects the quality of the learned knowledge. Another question is whether it is possible to escape from the region of the phase transition by suitably manipulating the control parameters. In this chapter we try to provide an answer to these questions, by means of an experimental analysis and its interpretation.

The experimental setting

In order to test the independence of the results from the learning algorithm, we used the learners FOIL (Quinlan and Cameron-Jones, 1993), SMART + (Botta and Giordana, 1993), G-Net (Anglano et al., 1997; Anglano and Botta, 2002), and PROGOL (Muggleton, 1995) described in Chapter 6.

The fluctuating-hydrodynamics approach discussed earlier takes into account only the transport properties at the level of completely uncorrelated motion of the fluid particles. The corresponding dissipative processes are expressed in terms of bare transport coefficients of the fluid. The strongly correlated motion of the fluid particles which occurs at high density is not take into consideration here. This is reflected through the Markov approximation of the transport coefficients and the short correlation of the corresponding noise representing the fast degrees of freedom in the system. The Markovian equations for the collective modes involving frequency-independent transport coefficients constitute a model for the dynamics of fluids with exponential relaxation of the fluctuations. The corresponding equations of motion for the collective modes are linear. However, exceptions occur in certain situations in which the description of the dynamics cannot be reduced to a set of linearly coupled fluctuating equations with frequency-independent transport coefficients. In this chapter we will consider the nonlinear dynamics of the hydrodynamic modes for studying the strongly correlated motion of the particles in a dense fluid.

Nonlinear Langevin equations

We present in this section the formulation of a set of nonlinear stochastic equations for the dynamics of the many-particle system. We first discuss the physical motivation for extension of the fluctuating-hydrodynamics approach to include nonlinear coupling of the slow modes.

In Chapter 9 we claimed that there is experimental evidence that in real-world applications also, where examples are not randomly generated, discriminant hypotheses found by relational learners lie on the phase transition edge. In order to support this claim, we discuss here the findings presented by Giordana and Saitta (2000) concerning two-real world applications. The first is a popular benchmark known as the mutagenesis dataset (Srinivasan et al., 1995), while the second is an application to mechanical troubleshooting in a chemical plant (Giordana et al., 1993). In both cases the learning problems were solved using G-Net, the relational learner based on evolutionary search described in Chapter 6 (Anglano et al., 1997, 1998).

It is worth noticing that datasets suitable for relational learning and available in public repositories are few and, in general, rather simple. In fact, the concept descriptions that have been learned from them contain few literals only and, mostly, two or three chained variables. The datasets that we present in this appendix are among the most complex approached with machine learning: for both, descriptions containing up to four variables and up to six binary relations have been discovered. For the sake of reference, Figure A.1 gives the same graph as Figure 9.9(a) but for n = 4. A phase transition is evident, but the expected complexity in the mushy region is much lower than that in Figure 9.9(a).

We have discussed the construction of the nonlinear Langevin equations for the slow modes in a number of different systems in the previous chapter. Next, we analyze how the nonlinear coupling of the hydrodynamic modes in these equations of motion affects the liquid dynamics. In particular, we focus here on the case of a compressible liquid in the supercooled region. In this book we will primarily follow an approach in which the effects of the nonlinearities are systematically obtained using graphical methods of quantum field theory. Such diagrammatic methods have conveniently been used for studying the slow dynamics near the critical point (Kawasaki, 1970; Kadanoff and Swift, 1968) or turbulence (Kraichnan 1959a, 1961a; Edwards, 1964). The present approach, which is now standard, was first described by Martin, Siggia, and Rose (1973). The Martin–Siggia–Rose (MSR) field theory, as this technique is named in the literature, is in fact a general scheme applied to compute the statistical dynamics of classical systems.

The field-theoretic method presented here is an alternative to the so-called memoryfunction approach. The latter in fact involves studying the dynamics in terms of non-Markovian linearized Langevin equations (see, for example, eqn. (6.1.1) which are obtained in a formally exact manner with the use of so called Mori–Zwanzig projection operators. This projection-operator scheme is described in Appendix A7.4). The generalized transport coefficients or the so-called memory functions in this case are frequency-dependent and can be expressed in terms of Green–Kubo forms of integrals of time correlation functions.

In the long journey undertaken in this book, we have visited statistical mechanics, constraint satisfaction problems and satisfiability, complex networks and natural systems, and, in particular, many facets of machine learning ranging from propositional to relational learning, grammatical inference, and neural networks. The thread that connects all these fields is the emergence of phenomena exhibiting sharp discontinuities. These phenomena are reminiscent of the phase transitions found in physics and, indeed, the methods of statistical physics have been employed with success to analyze them. In this chapter we try to summarize what we have learned from these connections and in particular from the role played by machine learning. Our aim is to point out gaps in the understanding of basic phenomena and to identify open questions that may suggest future research directions.

Phase transitions or threshold phenomena?

In a recent and very interesting paper, which recalls similar arguments put forwards in Percus et al. (2006), Zweig et al. (2010) have challenged the current view of phase transitions in computational problems, wondering whether the abrupt change observed in the probability of solution (the order parameter) in SAT problems is in fact nothing other than a “self-fulfilling” discontinuity, i.e., an existential discontinuity generated by the very definitions of the problem and of the order parameter.

The first argument in support of their claim is that it is easy to produce rather simple models that exhibit phase transition phenomena while, as most of us would agree, the essential ingredients that underly a “true” phase transition are lacking.