This chapter provides a survey of the common techniques for determining the sharp statistical and computational limits in high-dimensional statistical problems with planted structures, using community detection and submatrix detection problems as illustrative examples. We discuss tools including the first- and second-moment methods for analyzing the maximum-likelihood estimator, information-theoretic methods for proving impossibility results using mutual information and rate-distortion theory, and methods originating from statistical physics such as the interpolation method. To investigate computational limits, we describe a common recipe to construct a randomized polynomial-time reduction scheme that approximately maps instances of the planted clique problem to the problem of interest in total variation distance.

This chapter presents a game-theoretic solution to several challenges in electricity markets, e.g., intermittent generation; high levels of average prices; price volatility; and fundamental aspects concerning the environment, reliability, and affordability. It proposes a stochastic bi-level optimization model to find the optimal nodal storage capacities required to achieve a certain price volatility level in a highly volatile energy-only electricity market. The decision on storage capacities is made in the upper-level problem and the operation of strategic/regulated generation, storage, and transmission players is modeled in the lower-level problem using an extended stochastic (Bayesian) Cournot-based game.

Deep learning (DL) has seen tremendous recent successes in many areas of artificial intelligence. It has since sparked great interests in its potential use in power systems. However, success from using DL in power systems has not been straightforward. Even with the continuing proliferation of data collected in the power systems from, e.g., synchrophasors and smart meters, how to effectively use these data, especially with DL techniques, remains a widely open problem. This chapter shows that the great power of DL can be unleashed in solving many fundamentally hard high-dimensional real-time inference problems in power systems. In particular, DL, if used appropriately, can effectively exploit both the intricate knowledge from the nonlinear power system models and the expressive power of DL predictor models. This chapter also shows the great promise of DL in significantly improving the stability, resilience, and security of power systems.

We study compression for function computation of sources at nodes in a network at receiver(s). The rate region of this problem has been considered under restrictive assumptions. We present results that significantly relax these assumptions. For a one-stage tree network, we characterize a rate region by a necessary and sufficient condition for any achievable coloring-based coding scheme, the coloring connectivity condition. We propose a modularized coding scheme based on graph colorings to perform arbitrarily closely to derived rate lower bounds. For a general tree network, we provide a rate lower bound based on graph entropies and show that it is tight for independent sources. We show that, in a general tree network case with independent sources, to achieve the rate lower bound, intermediate nodes should perform computations, but for a family of functions and random variables, which we call chain-rule proper sets, it suffices to have no computations at intermediate nodes to perform arbitrarily closely to the rate lower bound. We consider practicalities of coloring-based coding schemes and propose an efficient algorithm to compute a minimum-entropy coloring of a characteristic graph.

Clustering is a general term for techniques that, given a set of objects, aim to select those that are closer to one another than to the rest, according to a chosen notion of closeness. It is an unsupervised-learning problem since objects are not externally labeled by category. Much effort has been expended on finding natural mathematical definitions of closeness and then developing/evaluating algorithms in these terms. Many have argued that there is no domain-independent mathematical notion of similarity but that it is context-dependent; categories are perhaps natural in that people can evaluate them when they see them. Some have dismissed the problem of unsupervised learning in favor of supervised learning, saying it is not a powerful natural phenomenon. Yet, most learning is unsupervised. We largely learn how to think through categories by observing the world in its unlabeled state. Drawing on universal information theory, we ask whether there are universal approaches to unsupervised clustering. In particular, we consider instances wherein the ground-truth clusters are defined by the unknown statistics governing the data to be clustered.

The increasing penetration of renewable resources has changed the characteristics of power system and market operations, from one relying primarily on deterministic and static planning to one involving highly stochastic and dynamic operations. In such new operation regimes, the ability of adapting changing environments and managing risks arising from complex scenarios of contingencies is essential. To this end, an operation tool that provides probabilistic forecasting that characterizes the underlying probability distribution of variables of interest can be extremely valuable. A fundamental challenge in probabilistic forecasting for system and market operations is the scalability. As the size of system and the complexity of stochasticity increase, standard techniques based on direct Monte Carlo and machine learning techniques become intractable. This chapter outlines an alternative approach based on an online learning to overcome barriers of computation complexity.

Information theory plays an indispensable role in the development of algorithm-independent impossibility results, both for communication problems and for seemingly distinct areas such as statistics and machine learning. While numerous information-theoretic tools have been proposed for this purpose, the oldest one remains arguably the most versatile and widespread: Fano’s inequality. In this chapter, we provide a survey of Fano’s inequality and its variants in the context of statistical estimation, adopting a versatile framework that covers a wide range of specific problems. We present a variety of key tools and techniques used for establishing impossibility results via this approach, and provide representative examples covering group testing, graphical model selection, sparse linear regression, density estimation, and convex optimization.

This chapter introduces basic ideas of information-theoretic models for distributed statistical inference problems with compressed data, and discusses current and future research directions and challenges in applying these models to various statistical learning problems. In these applications, data are distributed in multiple terminals, which can communicate with each other via limited-capacity channels. Instead of recovering data at a centralized location first and then performing inference, this chapter describes schemes that can perform statistical inference without recovering the underlying data. Information-theoretic tools are borrowed to characterize the fundamental limits of the classical statistical inference problems using compressed data directly. In this chapter, distributed statistical learning problems are first introduced. Then, models and results of distributed inference are discussed. Finally, new directions that generalize and improve the basic scenarios are described.

This chapter provides an overview of the theory of controlled sensing, and its application to the sequential design of data-acquisition and decision-making processes. Based on the theory, it provides an overview of the applications to the quickest detection and localization of anomalies in power systems. This application is motivated by the fact that agile localization of anomalous events plays a pivotal role in enhancing the overall reliability of the grid and avoiding cascading failures. This is especially of paramount significance in the large-scale grids due to their geographical expansions and the large volume of data generated. Built on the theory of controlled sensing, the chapter discusses a stochastic graphical framework for localizing the anomalies with the minimum amount of data. This framework capitalizes on the strong correlation structures observed among the measurements collected from different buses. This framework, at its core, collects the measurements sequentially and progressively updates its decision about the location of the anomaly.

This chapter reviews the methods used to estimate the state of a power system and its network model based on the measurements provided by supervisory control and data acquisition (SCADA) systems and/or by phasor measurement units (PMU). Initially, it provides an overview of the commonly implemented SCADA-based state estimators. Network observability and bad data processing functions are briefly described. This is followed by a description of the changes in the problem formulation and solution introduced by the incorporation of PMU measurements as well as the associated opportunities and challenges. Finally, detection, identification, and correction of network model errors and impact of such errors on system reliability as well as market operations are presented.

This chapter focuses on distributed control and learning for electric vehicle charging. After a brief survey, it covers three related sets of algorithms: (i) distributed control for electric vehicle charging based on a basic formulation; (ii) distributed control for an extension of the basic setting to include network capacity constraints; and (iii) distributed learning for an extension of the basic setting with limitations in the information flow. The chapter ends with a brief summary of open problems.

This chapter introduces mean field games to capture the mutual interaction between a population and its individuals.Within this context, a new equilibrium concept called mean field equilibrium replaces the classical Nash equilibrium in game theory. In a mean field equilibrium each individual responds optimally to the population behavior. In other words, no individuals have incentives to deviate from their current strategies. This new way of modeling the interactions among members of large populations is used to study dynamic demand response management in electricity grids. Moreover, some generalizations of the classical idea of mean field games are introduced to embrace the situations in which the whole population can be divided into classes of members.

The problem of tracking the system frequency is ubiquitous in power systems. However, despite numerous empirical comparative studies of various algorithms, the underlying links and commonalities between frequency tracking methods are often overlooked. To this end, we show that the treatment of the two best known frequency estimation methodologies: (i) tracking the rate of change of the voltage phasor angles, and (ii) fixed frequency demodulation, can be unified, whereby the former can be interpreted as a special case of the latter. Furthermore, we show that the frequency estimator derived from the difference in the phase angle is the maximum likelihood frequency estimator of a nonstationary sinusoid. Drawing upon the data analytics interpretation of the Clarke and related transforms in power system analysis as practical Principal Component Analyzers (PCA), we then set out to explore commonalities between classic frequency estimation techniques and widely linear modeling. The so-obtained additional degrees of freedom allow us to arrive at the adaptive Smart Clarke and Smart Park transforms (SCT and SPT), which are shown to operate in an unbiased and statistically consistent way for both standard and dynamically unbalanced smart grids. Overall, this work suggest avenues for next generation solutions for the analysis of modern grids that are not accessible from the Circuit Theory perspective.