In this book, we introduce a fundamental relation between algebraic geometry and statistical learning theory.

A lot of statistical models and learning machines used in information science, for example, mixtures of probability distributions, neural networks, hidden Markov models, Bayesian networks, stochastic context-free grammars, and topological data analysis, are not regular but singular, because they are nonidentifiable and their Fisher information matrices are singular. In such models, knowledge to be discovered from examples corresponds to a singularity, hence it has been difficult to develop a mathematical method that enables us to understand statistical estimation and learning processes.

Recently, we established singular learning theory, in which four general formulas are proved for singular statistical models. Firstly, the log likelihood ratio function of any singular model can be represented by the common standard form even if it contains singularities. Secondly, the asymptotic behavior of the evidence or stochastic complexity is clarified, giving the result that the learning coefficient is equal to the maximum pole of the zeta function of a statistical model. Thirdly, there exist equations of states that express the universal relation of the Bayes quartet. We can predict Bayes and Gibbs generalization errors using Bayes and Gibbs training errors without any knowledge of the true distribution. And lastly, the symmetry of the generalization and training errors holds in the maximum likelihood and a posteriori estimators. If one-point estimation is applied to statistical learning, the generalization error is equal to the maximum value of a Gaussian process on a real analytic set.