This chapter explores Bayesian optimization and statistics, which provides a flexible framework for balancing prior knowledge about a system with new data. Many optimization techniques are either rooted in Bayesian statistics, or have a Bayesian interpretation. For example, the Tikhonov and LASSO regularized least-squares come from specific prior distributions on the unknown parameters. Bayesian methods are particularly useful when we have small data sets or complex models. They are used extensively for hyperparameter tuning of machine learning models, for experimental design, and for applications in engineering that involve complex partial differential equations. This chapter provides a high-level overview of Bayesian methods in optimization, starting with statistical estimation. Density estimation, regularization, and expectation maximization will be explored here. Estimating a statistical distribution from data is fundamentally an optimization problem, and is the basis of machine learning. The simplest approach is to assume a known distribution and estimate the parameters using optimization. If we include Bayesian prior information, this becomes maximum a posteriori estimation.