The study of the emergence of phase transitions, or, more generally, the application of statistical physics methods to automated learning, is not new. For at least a couple of decades ensemble phenomena have been noticed in artificial neural networks. In the first part of this chapter we will illustrate these early results and then move to currently investigated issues.

According to Watkin et al. (1993), statistical physics tools are not only well suited to analyze existing learning algorithms but also they may suggest new approaches. In the paradigm of learning from examples (the paradigm considered in this book), examples are drawn from some unknown but fixed probability distribution and, once chosen, constitute a static quenched disorder (Watkin et al., 1993).

Artificial neural networks

Artificial neural networks (NNs) are graphs consisting of a set of nodes that correspond to elementary computational units, called “neurons”, connected via links recalling “axons” and “synapses”. Even though the terminology is borrowed from neurology, the analogy between an NN and the brain can well be ignored. The idea behind the computation in an NN is that inputs from the external world activate a subset of the neurons (the input neurons), which, in turn, elaborate the input and transmit the results of this local computation to other neurons until some subset of neurons (the output neurons) provide the final results to the external world.