Communications, Information Theory and Security

Summary

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]:

• They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised).
• They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation.
• They are part of the broader machine learning field of learning representations of data.
• They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.

These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks.

In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Summary

In this chapter, we discuss distributed algorithms for solving large-scale optimization problems over networks. In particular, the optimization problem is defined over a network of nodes/agents where each of them may have a piece of data, or may be in charge of updating a subset of variables, and they are required to collaboratively perform the desired optimization. The key question to be addressed is, under various problem settings, what protocols should the agents follow so as to efficiently optimize the overall problem. These types of distributed optimizing algorithms have recently become increasingly important in various big data problems, with applications ranging from distributed machine learning, signal processing, networking, multi-agent systems, parallel optimization, etc. There are a few driving forces behind these new trends, as we briefly list below:

1. The advances in sensor technologies makes distributed data acquisition ubiquitous.
2. The increasing size of optimization problems in big data era calls for distributed data storage.
3. Privacy issues in big data problem prevents direct data exchange among different (possibly geographically distributed) data sources.
4. Using multi-core high-performance computing systems (with either shared or distributed memory) to solve big data optimization problems can significantly reduce the solution time.

In this chapter, we present a few recent algorithms for solving such distributed optimization problems. We will cover a wide range of algorithms and discuss their convergence behavior, as well as their performance in terms of scaling with network/problem sizes.

The chapter is organized as follows. In Sections 9.1–9.4, we present a general problem formulation and a few popular first-order methods. In Section 9.5, we discuss a special cases of the general formulation (i.e., the global consensus problem). In Section 9.6, we compare various algorithms presented in this chapter.

Background

Problem Formulation

The Optimization Problem

Consider a network with N distributed agents, each capable of communicating with its immediate neighbors; see Figure 9.1 for an illustration.

Summary

This chapter reviews a collection of sparse optimization models and algorithms. The review focuses on introducing these algorithms along with their motivations and basic properties. The mathematical tools from the previous section will be heavily replied upon.

This chapter is organized as follows. Section 4.1 reviews a list of (convex) sparse optimization models, which deal with different types of signals and different kinds of noise; they can also include additional features as objectives and constraints that arise in practical problems. Section 4.2 demonstrates that the convex sparse optimization problems can be transformed in equivalent cone programs and solved by off-the-shelf algorithms; yet it argues that these algorithms are usually inefficient for large-scale problems. Sections 4.3–4.12 cover a large variety of algorithms for sparse optimization. These algorithms have different strengths and fit different applications. The efficiency of these algorithms, for the most part, comes from the use of the often closed-form shrinkage-like operations, which is introduced in Section 4.4. Then, Section 4.5 presents a prox-linear framework and gives several algorithms under this framework that are based on gradient descent and take advantages of shrinkage-like operations. These algorithms can often be linked with the BSUM framework discussed in Chapter 3. Duality is a very powerful tool in modern convex optimization; sparse optimization is not an exception. Section 4.6 derives a few dual models and algorithms for sparse optimization and discusses their properties. One class of dual algorithms (i.e., the ADMM algorithm introduced in Chapter 3) is very efficient and extremely versatile, applicable to nearly all convex sparse optimization problems. The framework and applications of these algorithms are given in Section 4.8. Unlike other algorithms, the homotopy algorithms in Section 4.9 produces not just one solution but a path of solutions for the LASSO model (which is given in (4.2b) below); not only are they efficient at producing multiple solutions corresponding to different parameter values, they are especially fast if the solution is extremely sparse. All the above algorithms can be (often significantly) accelerated by appropriately setting their stepsize parameters that determine the amount of progress at each iteration. Such techniques are reviewed in Section 4.10. Unlike all previous algorithms, greedy algorithms do not necessarily correspond to an optimization model; instead of systematically searching for solutions that minimize objective functions, they build sparse solutions by constructing their supports step by step in a greedy manner.

Summary

Introduction to spatially multiplexed MIMO systems

We have observed in chapter 2 that the capacity of wireless communication links is increased by using multiple antennas at the transmitter and the receiver. To achieve these capacities, a transmission scheme, called Diagonal-Bell Laboratories Layered Space-time (D-BLAST) has been proposed by G. J. Foschini (1996). In an i.i.d. Rayleigh scattering environment, this processing structure leads to theoretical rates which grow linearly with the number of antennas (for NR = NT ) with these rates approaching 90% of Shannon's capacity. But this has large computational complexity required for implementation of this scheme.

A simplified version, called Vertical BLAST (V-BLAST) has been proposed by G. J. Foschini et al., (1999). They have demonstrated spectral efficiencies of 20–40 bps/Hz at average signal-to-noise ratio (SNR) ranging from 24 to 34 dB could be achieved in indoor environments. The essential difference between D-BLAST and V-BLAST lies in the vector encoding process. The D-BLAST code blocks are organized along diagonals in space-time. In V-BLAST, however, the vector encoding process is simply a demultiplexing (DMUX) operation followed by independent bit-to-symbol mapping of each sub-stream. No inter sub-stream coding, or coding of any kind, is required.

Note that BLAST detection scheme can be done in one of the following ways:

Summary

Introduction

Since using SVD, we can decompose a MIMO channel into RH parallel Gaussian channels, where RH is the rank of the MIMO channel matrix, we will use the knowledge on the capacity of the parallel Gaussian channel (see Appendix C) to find the capacity of a MIMO channel for uniform and adaptive power allocation scheme. Uniform power allocation is employed when the channel state information (CSI) is available at the receiver but not at the transmitter (open loop MIMO system). We can use adaptive power allocation based on Water-filling algorithm when CSI is available at the receiver as well as the transmitter (closed loop MIMO system). We will also discuss near optimal power allocation for high and low SNR cases.

Note that power allocation plays a significant role in deciding MIMO capacity. Power allocation was not an important issue in SISO since only single antenna was employed at the transmitter and receiver. Usually we allocate all the power to the single transmit antenna. But for MIMO it is one of the most important factors for increasing capacity. We have numerous antennas at the transmitter and receiver for MIMO case. The fundamental question is how much power we allocate to each transmit antennas. Hence if we allot power equally to all transmit antennas or unequally to each transmit antenna, capacity of the MIMO channel will be definitely different. If this is the case, then how we optimally allocate power to MIMO channels can be considered as an optimization problem to maximize capacity. To allocate power adaptively we need the CSI at the transmitter, also since power allocation is done at the transmitter. Intuitively we will allocate more power to better channels than the bad channels. We may not allocate any power at all to some of the worst channels. We will discuss these in detail in the following sections. In practical scenarios, we can allocate power near optimally for MIMO channels for two cases: high and low SNR regimes.

Uniform power allocation

The capacity indicates the best viable transmission data rate over the channel for miniscule probability of error. Shanon provided the expression of the achievable communication rate of a channel with noise (C. E. Shanon, 1948).

Summary

Introduction

Game theory presents a formal analysis of interaction among a group of players in a game. In game theory, the action of player directly affects the other players. There are two types of games:

Communications, Information Theory and Security

Refine search

Refine search

Actions for selected content:

1455 results in Communications, Information Theory and Security

Contents

Frontmatter

10 - Optimization of Finite Sums

Part II - Methodology and Mathematical Background

7 - Deep Learning and Applications

Summary

9 - Distributed Large-Scale Optimization

Summary

Bibliography

4 - Sparse Optimization

Summary

10 - Advanced MIMO Detection Techniques

Summary

7 - Introduction to Space-Time Codes

9 - Introduction to MIMO Detection

Appendix B - MGF of Hermitian Quadratic Form in Complex Gaussian Variate

Appendix F - Algebraic Structures

Contents

12 - Advanced Topics in MIMO Wireless Communications

Appendix A - Matrices

Acknowledgments

4 - Power Allocation in MIMO Systems

Summary

Appendix G - An Introduction to Game Theory

Summary

Preface

Communications, Information Theory and Security

Refine search

Refine search

Actions for selected content:

Save Search

1455 results in Communications, Information Theory and Security

Summary

Summary

Summary

Summary

Summary

Summary