Inference and Learning from Data Inference

We formulated the maximum-likelihood (ML) approach in the previous chapter, where an unknown parameter θ is estimated by maximizing the log-likelihood function. We showed there that in some cases of interest this problem can be solved analytically in closed form and an expression for the parameter estimate can be determined in terms of the observations. However, there are many important scenarios where the ML solution cannot be pursued in closed form, either due to mathematical intractability or due to missing data or hidden variables that are unobservable. In this chapter, we motivate and describe the expectation maximization (EM) procedure as a useful tool for constructing ML estimates under these more challenging conditions. We also illustrate how EM can be used to fit mixture models onto data.