AI systems have struggled to be deployed in safety-critical applications where the consequences of incorrect predictions are severe. In complex applications and environments, like autonomous driving, it is often impossible or impractical to curate a dataset or simulator that sufficiently spans the entire input space, making it improbable that a perfect agent can be trained offline. Metacognitive AI represents an approach to design agents that continue safely learning and adapting as they encounter new or uncertain scenarios in the environment, which improves their performance over time. A key component to achieve this behavior is quantifying the AI agent’s prediction uncertainty to enable the agent to understand when it is operating in a previously unseen scenario. In this chapter, we discuss a framework for creating a metacognitive agent and delve deeper into Meta Modeling, a method for augmenting existing neural networks with uncertainty quantification. Our approach provides a first step toward realizing a metacognitive AI agent.