Under time series analysis, one proceeds from Fourier analysis to the design of windows, then spectral analysis (e.g. computing the spectrum, the cross-spectrum between two time series, wavelets, etc.) and the filtering of frequency signals. The principal component analysis method can be turned into a spectral method known as singular spectrum analysis. Auto-regressive processes and Box-Jenkins models are also covered.

As probability distributions form the cornerstone of statistics, a survey is made of the common families of distributions, including the binomial distribution, Poisson distribution, multinomial distribution, Gaussian distribution, gamma distribution, beta distribution, von Mises distribution, extreme value distributions, t-distribution and chi-squared distribution. Other topics include maximum likelihood estimation, Gaussian mixtures and kernel density estimation.

Inspired by the human brain, neural network (NN) models have emerged as the dominant branch of machine learning, with the multi-layer perceptron (MLP) model being the most popular. Non-linear optimization and the presence of local minima during optimization led to interests in other NN architectures that only require linear least squares optimization, e.g. extreme learning machines (ELM) and radial basis functions (RBF). Such models readily adapt to online learning, where a model can be updated inexpensively as new data arrive continually. Applications of NN to predict conditional distributions (by the conditional density network and the mixture density network) and to perform quantile regression are also covered.

A review of basic probability theory – probability density, expectation, mean, variance/covariance, median, median absolute deviation, quantiles, skewness/kurtosis and correlation – is first given. Exploratory data analysis methods (histograms, quantile-quantile plots and boxplots) are then introduced. Finally, topics including Mahalanobis distance, Bayes theorem, classification, clustering and information theory are covered.

Simple linear regression is extended to multiple linear regression (for multiple predictor variables) and to multivariate linear regression for (multiple response variables). Regression with circular data and/or categorical data is covered. How to select predictors and how to avoid overfitting with techniques such as ridge regression and lasso are followed by quantile regression. The assumption of Gaussian noise or residual is removed in generalized least squares, with applications to optimal fingerprinting in climate change.

The historical development of statistics and artificial intelligence (AI) is outlined, with machine learning (ML) emerging as the dominant branch of AI. Data science is viewed as being composed of a yin part (ML) and a yang part (statistics), and environmental data science is the intersection between data science and environmental science. Supervised learning and unsupervised learning are compared. Basic concepts of underfitting/overfitting and the curse of dimensionality are introduced.

From observed data, statistical inference infers the properties of the underlying probability distribution. For hypothesis testing, the t-test and some non-parametric alternatives are covered. Ways to infer confidence intervals and estimate goodness of fit are followed by the F-test (for test of variances) and the Mann-Kendall trend test. Bootstrap sampling and field significance are also covered.

Bayesian optimization is a methodology for optimizing expensive objective functions that has proven success in the sciences, engineering, and beyond. This timely text provides a self-contained and comprehensive introduction to the subject, starting from scratch and carefully developing all the key ideas along the way. This bottom-up approach illuminates unifying themes in the design of Bayesian optimization algorithms and builds a solid theoretical foundation for approaching novel situations.

The core of the book is divided into three main parts, covering theoretical and practical aspects of Gaussian process modeling, the Bayesian approach to sequential decision making, and the realization and computation of practical and effective optimization policies.

Following this foundational material, the book provides an overview of theoretical convergence results, a survey of notable extensions, a comprehensive history of Bayesian optimization, and an extensive annotated bibliography of applications.

Bayesian optimization is a methodology for optimizing expensive objective functions that has proven success in the sciences, engineering, and beyond. This timely text provides a self-contained and comprehensive introduction to the subject, starting from scratch and carefully developing all the key ideas along the way. This bottom-up approach illuminates unifying themes in the design of Bayesian optimization algorithms and builds a solid theoretical foundation for approaching novel situations.

The core of the book is divided into three main parts, covering theoretical and practical aspects of Gaussian process modeling, the Bayesian approach to sequential decision making, and the realization and computation of practical and effective optimization policies.

Following this foundational material, the book provides an overview of theoretical convergence results, a survey of notable extensions, a comprehensive history of Bayesian optimization, and an extensive annotated bibliography of applications.

Bayesian optimization is a methodology for optimizing expensive objective functions that has proven success in the sciences, engineering, and beyond. This timely text provides a self-contained and comprehensive introduction to the subject, starting from scratch and carefully developing all the key ideas along the way. This bottom-up approach illuminates unifying themes in the design of Bayesian optimization algorithms and builds a solid theoretical foundation for approaching novel situations.

The core of the book is divided into three main parts, covering theoretical and practical aspects of Gaussian process modeling, the Bayesian approach to sequential decision making, and the realization and computation of practical and effective optimization policies.

Following this foundational material, the book provides an overview of theoretical convergence results, a survey of notable extensions, a comprehensive history of Bayesian optimization, and an extensive annotated bibliography of applications.