2.1.1. Markov decision process

The elements of an Reinforcement Learning (RL) problem, as described in Section 2.1 can be formalized using the Markov Decision Process (MDP) framework (Puterman, Reference Puterman1994; Boutilier et al., Reference Boutilier, Dean and Hanks1999). Markov Decision Process (MDP) can efficiently model the interactions between the environment and the agent by defining a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}\rangle$ , where:

• • $\mathcal{S}$ represents the finite set of states, $s_{t} \in \mathcal{S}$ with dimensionality N, that is, $|\mathcal{S}|=N$ . Recall, a state is a distinctive characteristic of the environment or the problem being modeled.

• • $\mathcal{A}$ represents the finite set of actions, $a_{t} \in \mathcal{A}$ with dimensionality K, that is, $|\mathcal{A}|=K$ . Each action $a_t$ is used as a control to interact with the environment at time t. A set of available actions at a particular state s is denoted by A(s), where $A(s) \subseteq \mathcal{A}$ .

• • $ \mathcal{P}\,{:}\, \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow[0,1]$ is the stochastic transition function, where $p\left(s^{\prime} \mid s, a\right)$ , also denoted as $p_{s, s^{\prime}}^a$ is the probability of transitioning to state $s^{\prime}$ by taking action a at state s. These transition probabilities can also be stationary, that is, they are independent of time t.

(2) \begin{equation} p\left(s^{\prime} \mid s, a\right)=p_{s, s^{\prime}}^a=\mathbb{P}\!\left\{s_{t+1}=s^{\prime} \mid s_{t}=s, a_{t}=a\right\}. \end{equation}

For this transition function to represent a proper probability distribution the following properties must hold:

(3) \begin{equation} p\!\left(s^{\prime} \mid s, a\right) \geq 0 \text { and } p\!\left(s^{\prime} \mid s, a\right) \leq 1 \end{equation}

(4) \begin{equation} \sum_{s^{\prime} \in S} p\!\left(s^{\prime} \mid s, a\right)=1 \end{equation}

Since MDPs are controlled Markov chains (Markov, Reference Markov1906; Markov, Reference Markov2006), the distribution of state at time $t+1$ is independent of the past given the state at time t and the action performed by the agent.

(5) \begin{align} \mathbb{P}\!\left[s_{t+1}=s^{\prime} \mid s_{t}, a_{t}, s_{t-1}, a_{t-1}, \ldots\right]=\mathbb{P}\!\left[s_{t+1}=s \mid s_{t}, a_{t}=a\right] \\[-20pt] \nonumber \end{align}

• • $\mathcal{R}\,{:}\, \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the state reward function. It is a scalar value, $r\!\left(s, a, s^{\prime}\right)$ , also denoted as $r_{s, s^{\prime}}^a$ , which is awarded to the agent for transitioning from state s to $s^\prime$ by performing an action a.

  (6) \begin{equation} r\!\left(s, a, s^{\prime}\right)=\mathcal{R}_{s, s^{\prime}}^a=\mathbb{E}\!\left\{R_{t+1}\mid s_{t}=s, a_{t}=a,s_{t+1}=s^{\prime}\right\}. \end{equation}
  Alternatively, rewards can also be defined as $R\,{:}\, S \times A \rightarrow \mathbb{R}$ where a reward is awarded for taking an action in a state.
  (7) \begin{equation} r\!\left(s, a\right)=t_{s, a}=\mathbb{E}\!\left\{R_{t+1}\mid s_{t}=s, a_{t}=a\right\}. \end{equation}

2.1.2. Policies and optimality criteria

In the previous section, we have defined a mathematical formalism for RL problems using an MDP. To formally determine the expected return, $G_t$ that is, objective that the policy seeks to optimize, two classes of MDPs are discussed:

• • Finite Horizon : Given a fixed time horizon of T, the objective is to find a sequential decision policy $\pi$ , that maximizes the expected return until the end of the time horizon. If we represent the sequence of rewards received after time t, by $r_{t+1}, r_{t+2}, r_{t+3}, \ldots$ , then the cumulative reward is simply:

  (8) \begin{equation} G_{t} \doteq r_{t+1}+r_{t+2}+r_{t+3}+\cdots+r_{T} \end{equation}

• • Infinite Horizon—Discounted rewards : Given an endless time horizon $T=\infty$ , the objective is to find a sequential decision policy, $\pi$ that maximizes the expected discounted return.

  (9) \begin{equation} G_{t} \doteq r_{t+1}+\lambda r_{t+2}+\lambda^{2} r_{t+3}+\cdots=\sum_{l=0}^{\infty} \lambda^{l} r_{t+l+1} \end{equation}
  where $\lambda$ is a parameter, $0 \leq \lambda<1$ , which discounts the future rewards, the rewards received after l time steps are worth $\lambda^{l-1}$ times their original value. The discounting of rewards also ensures that the expected return converges. The following recursive relation between the rewards of successive time steps is the key result used in several RL algorithms.
  (10) \begin{align} G_{t} & \doteq r_{t+1}+\lambda r_{t+2}+\lambda^{2} r_{t+3}+\lambda^{3} r_{t+4}+\cdots \nonumber \\[3pt] & =r_{t+1}+\lambda\!\left(r_{t+2}+\lambda r_{t+3}+\lambda^{2} r_{t+4}+\cdots\right) \nonumber\\[3pt] & =r_{t+1}+\lambda G_{t+1} \\[-30pt] \nonumber \end{align}

2.1.3. Value functions and Bellman equation

A stochastic policy $\pi$ , is a mapping from a state $s\in \mathcal{S}$ to action $a\in\mathcal{A}(s)$ . It represents the probability of taking action a in state s, $\pi(a \mid s)$ . The state-value function, or simply value function, for a state s under a policy $\pi$ is denoted as $v_{\pi}(s)$ . It is the expected reward that can be accumulated starting with state s and following the policy $\pi$ thereafter. For an infinite horizon model as described in the previous section, the state-value function can be formalized as:

(11) \begin{equation} v_{\pi}(s) \doteq \mathbb{E}_{\pi}\!\left[G_{t} \mid S_{t}=s\right]=\mathbb{E}_{\pi}\!\left[\sum_{l=0}^{\infty} \lambda^{l} R_{t+l+1} \mid S_{t}=s\right]\end{equation}

Similarly, state-action-value function $Q\,{:}\, S \times A \rightarrow \mathbb{R}$ is defined as the expected reward that can be gathered starting with state s, taking an action a and following the policy $\pi$ thereafter.

(12) \begin{equation} q_{\pi}(s, a) \doteq \mathbb{E}_{\pi}\!\left[G_{t} \mid S_{t}=s, A_{t}=a\right]=\mathbb{E}_{\pi}\!\left[\sum_{l=0}^{\infty} \lambda^{l} R_{t+l+1} \mid S_{t}=s, A_{t}=a\right]\end{equation}

A recursive formulation of Equation (11) which relates the value function of a state ( $v_{\pi}(s)$ ) to the value function of its successor states ( $v_{\pi}(s')$ ) is a fundamental result utilized throughout RL. It is known as the Bellman equation (Bellman, Reference Bellman1958):

(13) \begin{equation} \begin{aligned} v_{\pi}(s) & \doteq \mathbb{E}_{\pi}\!\left[G_{t} \mid S_{t}=s\right] \\[3pt] & =\mathbb{E}_{\pi}\!\left[r_{t+1}+\lambda G_{t+1} \mid S_{t}=s\right] \\[3pt] & =\sum_{a} \pi(a \mid s) \sum_{s^{\prime}} \sum_{r} p\!\left(s^{\prime}, r \mid s, a\right)\!\left[r+\lambda \mathbb{E}_{\pi}\!\left[G_{t+1} \mid S_{t+1}=s^{\prime}\right]\right] \\[3pt] & =\sum_{a} \pi(a \mid s) \sum_{s^{\prime}, r} p\!\left(s^{\prime}, r \mid s, a\right)\!\left[r+\lambda v_{\pi}\!\left(s^{\prime}\right)\right] \end{aligned}\end{equation}

A policy $\pi$ is considered better than $\pi^{\prime}$ iff $v_{\pi}(s) \geq v_{\pi^{\prime}}(s)$ for all states $s \in \mathcal{S}$ . The policy which is better than all is known as the optimal policy: $\pi_{*}$ , its value function, denoted by $v_{*}$ , can be formulated as:

(14) \begin{equation} \begin{array}{l} \qquad v_{*}(s) \doteq \max _{\pi} v_{\pi}(s), \text { for all } s \in \mathcal{S}. \end{array}\end{equation}

The optimal state-action-value function, denoted by $q_{*}$ , can be formulated as:

(15) \begin{equation} q_{*}(s, a) \doteq \max _{\pi} q_{\pi}(s, a)\end{equation}

The Bellman optimality equation declares that the value function of a state under the optimal policy is the expected return received from the ‘best (greedy)’ action in that state.

(16) \begin{equation} \begin{aligned} v_{*}(s) & =\max _{a \in \mathcal{A}(s)} q_{\pi_{*}}(s, a) \\[3pt] & =\max _{a} \mathbb{E}_{\pi_{*}}\!\left[G_{t} \mid S_{t}=s, A_{t}=a\right] \\[3pt] & =\max _{a} \mathbb{E}_{\pi_{*}}\!\left[r_{t+1}+\lambda G_{t+1} \mid S_{t}=s, A_{t}=a\right] \\[3pt] & =\max _{a} \mathbb{E}\!\left[r_{t+1}+\lambda v_{*}\!\left(S_{t+1}\right) \mid S_{t}=s, A_{t}=a\right] \\[3pt] & =\max _{a} \sum_{s^{\prime}, r} p\!\left(s^{\prime}, r \mid s, a\right)\!\left[r+\lambda v_{*}\!\left(s^{\prime}\right)\right]. \end{aligned}\end{equation}

Similarly, the bellman optimality equation for $q_{*}$ , can be formulated as:

(17) \begin{equation} \begin{aligned} q_{*}(s, a) & =\mathbb{E}\!\left[r_{t+1}+\lambda \max _{a^{\prime}} q_{*}\!\left(S_{t+1}, a^{\prime}\right) \mid S_{t}=s, A_{t}=a\right] \\[3pt] & =\sum_{s^{\prime}, r} p\!\left(s^{\prime}, r \mid s, a\right)\!\left[r+\lambda \max _{a^{\prime}} q_{*}\!\left(s^{\prime}, a^{\prime}\right)\right] \end{aligned}\end{equation}

2.2.1. Dynamic programming

When the model for the problem is known, dynamic programming (Bellman, Reference Bellman1954), a mathematical optimization technique, can be utilized to learn the optimal policy. Generalized Policy Iteration (GPI) is such an approach, and it uses an iterative two-step procedure to learn the optimal policy.

1. 1. Policy evaluation: The first step evaluates the state-value function $v_{\pi}$ using the recursive formulation in Equation (13).

2. 2. Policy improvement: The second step uses the value function to generate an improved policy $\pi' \geq \pi$ using a greedy approach formulated in Equation (17).

Since the improved policy $\pi'$ is greedy in actions to the current policy, that is, $\pi'(s) = \arg\max_{a \in \mathcal{A}} q_\pi(s, a)$

(18) \begin{equation} \begin{aligned} q_\pi(s, \pi'(s)) & = q_\pi(s, \arg\max_{a \in \mathcal{A}} q_\pi(s, a)) \\[3pt] & = \max_{a \in \mathcal{A}} q_\pi(s, a) \geq q_\pi(s, \pi(s)) = v_\pi(s) \end{aligned}\end{equation}

this iterative process is guaranteed to converge to the optimal value (Sutton & Barto, Reference Sutton and Barto2018).

2.2.2. Monte Carlo techniques

Monte Carlo techniques estimate the value functions without the need of system dynamics of the environment making it a model-free method. They rely on complete episodes to approximate the expected return, $G_{t}$ of a state. Recall, the value function, $v_t(s) = \mathbb{E}[ G_t \vert S_t=s]$ . In an episode, $s_{0}, a_{0}, r_{1}, s_{1}, a_{1}, r_{2}, s_{2}, a_{2}, r_{3}, \ldots$ , the occurrence of a state s is known as the visit to s. First-Visit Monte Carlo algorithm estimates the value function as the average of the returns ( $G_t$ ) following the first visits to s whereas Every-Visit Monte Carlo algorithm estimates it as the average of the returns following every visit to s.

The optimal policy is then evaluated using the same GPI explained previously. The schematic is shown in Figure 3.

Figure 3. GPI iterative process for optimal policy

2.2.3. Temporal-difference methods

Temporal Difference (TD) methods are model-free. Unlike MC methods, where the agents wait until the end of the episode for the final return to estimate the value functions, TD methods rely on an estimated final return, $r_{t+1} + \lambda v(s_{t+1})$ known as the TD target. The state-value function and state-action-value function are estimated at every step using the below formulation:

(19) \begin{equation} \begin{aligned} v(s_t) & \leftarrow v(s_t) + \alpha (r_{t+1} + \lambda v(s_{t+1}) - v(s_t)) \end{aligned}\end{equation}

(20) \begin{equation} \begin{aligned} q\!\left(s_{t}, a_{t}\right) & \leftarrow q\!\left(s_{t}, a_{t}\right)+\alpha\!\left(r_{t+1}+\lambda q\!\left(S_{t+1}, a_{t+1}\right)-q\!\left(s_{t}, a_{t}\right)\right) \end{aligned}\end{equation}

where $\alpha$ is a hyperparameter called the learning rate, which controls the rate at which the value gets updated at each step. This technique is called bootstrapping because the target value is updated using the current estimate of the return ( $r_{t+1} + \lambda v(s_{t+1})$ ) and not the exact return, $G_t$ .

2.2.4. Policy gradient

The methods discussed so far estimate the state-value, state-action-value functions respectively to learn what actions to take to maximize the overall return. Policy gradient consists of a family of methods that learn a parameterized policy without computing these value functions. A stochastic policy, parameterized with a set of parameters, $\theta \in \mathbb{R}^{d}$ , can be represented as, $\pi(a \mid s, \theta)=p\!\left(A_{t}=a \mid S_{t}=s, \boldsymbol{\theta}_{t}=\theta\right)$ , that is, the probability of taking an action at given state with a set of parameters $\theta$ at time t.

The parameters are learned by computing the gradient of an objective function, $J(\theta)$ , a scalar quantity that measures the performance of the policy. The goal of these methods is to maximize the performance by updating the parameters of the policy using stochastic gradient ascent (Bottou, Reference Bottou1998) as

(21) \begin{equation} \theta_{t+1}=\theta_{t}+\left.\alpha \nabla_{\theta} J\!\left(\pi_{\theta}\right)\right|_{\theta_{t}}\end{equation}

where $\nabla_{\theta} J\!\left(\pi_{\theta}\right) \in \mathbb{R}^{d}$ is an estimate whose expectation is equivalent to the gradient of the policy’s performance, known as policy gradient.

2.2.5. Actor-critic methods

They consist of algorithms that rely on estimating the parameters of the value functions (state-value or state-action-value function) concurrently with the policy. The two components include:

1. 1. Actor: is responsible for the agent’s policy, that is, it is used to pick out which action to perform. The underlying objective is to update the parameters of the policy using Equation (21) guided by the estimates from the critic as the baseline.

2. 2. Critic: is responsible for estimating the agent’s value functions. The objective is to update the parameters, $\phi$ , of the state-value function, $v(s;\ \phi)$ , or the state-action-value function, $q(s,a;\ \phi)$ by computing the TD error, $\delta_t$ as

   (22) \begin{equation} \delta_{t}=r_{t+1}+\lambda v\!\left(s_{t+1}\right)-v(s_{t}) \end{equation}
   Figure 4 shows the working of an actor-critic method.
   

   Figure 4. Architecture for actor-critic methods (Sutton & Barto, Reference Sutton and Barto2018)

2.3.1. Mathematical model

A NN is a computational model that consists of an input layer, hidden layers (any number), and an output layer. Each layer consists of several artificial neurons, the fundamental units of the network, which define a function on the inputs. Any network that uses several hidden layers (usually more than 3) is referred to as a Deep Neural Network (DNN), and the paradigm of ML that deals with DNN is known as Deep Learning (DL) (Goodfellow et al., Reference Goodfellow, Bengio and Courville2016).

In a feed-forward network, the interconnections between the nodes, that is, the neurons, do not form a cycle. Network topologies where the connections between neurons include a directed graph over a temporal sequence are known as Recurrent Neural Networks (RNNs). In this work, we primarily focus on a feed-forward network which is trained by performing a forward pass, also referred as forward propagation and a subsequent backward pass, also known as backpropagation.

Forward Propagation: When training a NN, the forward pass is the calculation phase, sometimes referred to as the evaluation phase. The input signals are forward propagated through all the neurons in each network layer in a sequential manner. The activation of a single neuron is shown in Figure 5, where the input signals $x_1,x_2\ldots x_n$ are first multiplied by a set of weights, $w_{i,j}$ , and then combined with a bias term, $b_j$ to obtain a pre-activation signal, $z_j$ . The bias is usually utilized to offset the pre-activation signal in case of an all-zero input; at times, this can alternatively be seen as an additional weight that is applied to an input, that is, a constant 1.

Figure 5. Activation of a single neuron in the neural network

An activation function, $g_j$ is then applied to $z_j$ to obtain the final output signal of the neuron. The process is then repeated across all neurons of each layer one by one until the final output is received. Figure 6 shows the forward propagation in a NN with a single hidden layer of 3 neurons. The final output, $x_k$ , is the prediction made by the NN given the input. This completes the forward pass of the network.

Figure 6. Forward propagation in an artificial neural network with a single hidden layer

Backpropogation: When a neural network is created, the parameters, $\theta$ including the weights and biases, are often initialized randomly or else sampled from a distribution. The prediction of the NN from the forward pass is compared with the true output, computing an error, E, using a loss function. The resultant scalar value indicates how well the parameters of the NN perform the specific task.

Training a NN involves adjusting the parameters of network $\theta$ , by first computing the derivative of the loss function and then using that information to update the parameters in the direction where the error is reduced. This is called gradient descent. Formally, the parameters are updated as:

(23) \begin{equation} \theta_{L}^{t+1}=\theta_{L}^{t}-\alpha \frac{\partial E}{\partial \theta_{L}^{t}}\end{equation}

where $\alpha$ is the learning rate. The term $\frac{\partial E}{\partial \theta_{L}^{t}}$ represents the partial derivative of the error with respect to the parameters of a layer L. These gradients can be computed by applying chain rule (shown in Figure 7) and involves the gradients of subsequent neurons and layers. Repeated application of chain rule can be used to calculate these gradients at every layer, all the way from the final layer, by navigating through the network backward, hence the name backpropogation (Rumelhart et al., Reference Rumelhart, Hinton and Mcclelland1986). This is a particular case of Automatic Differentiation (Rall, Reference Rall1981).

Figure 7. Backpropagation of gradients by application of chain rule

A cost function is another name for a loss function when applied to either the entire training data or a smaller subset known as the batch or mini-batch. Some commonly used cost functions include mean squared error, mean absolute error, cross-entropy, hinge loss, Kullback-Leibler divergence (Kullback & Leibler, Reference Kullback and Leibler1951) etc.

2.5.1. Genetic algorithm

GAs, first developed in 1975 (Holland, Reference Holland1992a), is a widely used class of evolutionary algorithms (Mitchell, Reference Mitchell1996) that takes inspiration from Charles Darwin’s theory of evolution (Darwin, Reference Darwin1859). It is meta-heuristic-based optimization technique that follows the principle of ‘survival of the fittest’.

The key components of any GA (Mitchell, Reference Mitchell1996) are:

• • Population: is the subset of all possible solutions to a problem. Every GA starts with an initialization of a set of solutions that become the first population. Each solution, individual from the population, is known as a chromosome which consists of several genes.

• • Genotype: is the representation of the individuals of a population in the computation space. This is done to ensure efficient processing and manipulation by the computing system.

• • Phenotype: is the actual representation of the population in the space of solutions.

• • Encoding and Decoding: For more straightforward problems, the genotype and the phenotype may be the same. However, complex problems rely on a translation mechanism to encode a phenotype to a genotype and a decoder for the reverse task.

• • Fitness function: is a tool to measure the performance of an individual on a specific problem.

• • Genetic operators: are tools utilized by the algorithm to produce a new set of solutions from the existing ones.

The algorithm starts by initializing a set of solutions, the population, which is also known as a generation. At each iteration, the fitness function is used to evaluate the performance of the entire population. The solutions with higher fitness (Baluja & Caruana, Reference Baluja and Caruana1995) are stochastically selected as the parents of the current generation. Genetic operators are applied to the parents to generate the next generation. This process is repeated until the required level of fitness is achieved. The complete workflow of a simple GA is shown in Figure 10.

Figure 10. Workflow of a simple genetic algorithm

The genetic operators are fundamental components of this algorithm that guides it toward a solution. The widely used genetic operators are summarized in the following subsections.

Selection Operators: The selection operator uses the fitness values as a measure of performance and picks a subset of the population as the current generation’s parents. A common strategy is to assign a probability value proportional to the fitness scores, with the goal that more fit individuals have a higher chance of being selected. Some other strategies include random selection, tournament selection, and roulette wheel selection, proportionate selection, steady-state selection, etc. (Haupt & Haupt, Reference Haupt and Haupt2003). The best performing individual from the current generation is selected and carried to the next generation without any alterations. This strategy is known as elitism (Baluja & Caruana, Reference Baluja and Caruana1995) and ensures that performance does not decrease over iterations.

Crossover operator: This operator picks a pair from the solutions shortlisted by the selection operator. The selected pair then mates and generates new solutions, which are known as the offspring. Each offspring is produced stochastically and contains genetic information, that is, features from either parent which is analogous to the crossover of genes in biology.

Single-point crossover is a commonly used crossover operator. A random crossover point is picked on the encoding of both parents. The offspring are then produced by recombination of the left partition of one parent with the right split of the other and vice versa. The same strategy can be extended to more than one crossover point. Figure 11 shows single-point crossover and two-point crossover between two binary encoded parents. In situations, where the encoding is an ordered list (Larranaga et al., Reference Larranaga1999), the solutions generated with the methods mentioned above, may be invalid. To overcome such issues, specialized operators like order crossover (Larranaga et al., Reference Larranaga1996), cycle crossover (CX) (Larranaga et al., Reference Larranaga1999) and partially mapped crossover have been published.

Figure 11. Single-point crossover (top) and two-point crossover (bottom) between two parents to generate offsprings for the next generation

Mutation operator: The mutation operator makes small arbitrary changes to an individual or its encoded representation which promotes the algorithm to explore the search space and find new solutions. As a result, the diversity within the population is improved during successive generations, and the algorithm can avoid getting stuck at local minima. Some commonly used mutation operators include Gaussian operators, uniform operators, bit flip operators (binary encoding), and shrink operators (Da Ronco & Benini, Reference Da Ronco, Benini and Kim2014). These operators are applied with a relatively small probability known as the mutation rate and have a few hyperparameters that can be tuned to specific problems.

2.5.2. Evolution strategies

Evolution Strategy (ES) is a black-box optimization technique that draws inspiration from the theory of natural selection, a process in which individuals from a population adapt to the changes of the environment.

Consider a 2D function, $f\,{:}\, \mathbb{R}^{2} \rightarrow \mathbb{R}$ , for which the gradient, $\nabla f\,{:}\, \mathbb{R}^{n} \rightarrow \mathbb{R}^{n}$ , can not be computed directly, and we wish to obtain the parameters $x^*$ and $y^*$ , such that $f(x^*,y^*)$ is the global maximum. The ES algorithm starts with a set of solutions for the problem, which are sampled from a parameterized distribution. An objective function then evaluates each solution and returns a single value as a fitness metric which is then utilized to update the parameters of the distribution.

Gaussian Evolution Strategy: One of the simplest techniques is to sample from a normal distribution, $\mathcal{N}\!\left(\mu, \sigma^{2}\right)$ , where $\mu$ is the mean and $\sigma^2$ is the variance. For the function defined above, one could start with $\mu=(0,0)$ and some constant, $\sigma=(\sigma_x,\sigma_y)$ . After one generation, the mean can be updated to the best performing solution from the population. Such a naive approach is only applicable to simple problems and also prone to getting stuck at local minima since it is greedy with respect to the best solution at each step of the training process.

Covariance Matrix Adaptation Evolution Strategy: The simple Gaussian strategy explained above assumes a constant variance. In some applications, it is worthwhile to explore the search space by having a larger variance initially and fine-tuning it later on when we are close to a good solution. Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) samples the solutions from a multivariate Gaussian distribution. It then adapts both the mean as well as the covariance matrix from the fitness metrics of the solutions. This algorithm is widely applicable to non-convex and nonlinear optimization problems (Hansen & Auger, Reference Hansen and Auger2011), especially in the continuous domain. There is no need to tune the parameters for a specific application since optimizing the parameters is incorporated within the design of this algorithm.

Natural Evolution Strategy: Simple Gaussian ES and CMA-ES both have the identical drawback of utilizing only the best individuals from the population and discarding all the others. At times, it might be beneficial to incorporate the information from low-scoring individuals as well, since they contain the vital information of what not to do. Natural Evolution Strategy (NES) (Wierstra et al., Reference Wierstra2008) utilize the information from the entire population to estimate the gradients, which are utilized to update the distribution from which the solutions were sampled. For an objective function, F with parameters $\theta$ which are sampled from a parameterized distribution $p_\psi(\theta)$ . The algorithm’s goal is to optimize the average objective $\mathbb{E}_{\theta \sim p_{\psi}}[F(\theta)]$ . The parameters of the distribution are iteratively updated using stochastic gradient ascent. The gradient of the objective is computed as follows:

(30) \begin{align} \nabla_{\psi}\!\left(\mathbb{E}_{\theta \sim p_{\psi}}[F(\theta)]\right) & =\nabla_{\psi}\!\left(\int_{\theta} p_{\psi}(\theta) \cdot F(\theta) \cdot d \theta\right) \nonumber\\ & =\int_{\theta} \nabla_{\psi}\!\left(p_{\psi}(\theta)\right) \cdot F(\theta) \cdot d \theta \nonumber\\ &=\int_{\theta} p_{\psi}(\theta) \cdot \nabla_{\psi}\!\left(\log p_{\psi}(\theta)\right) \cdot F(\theta) \cdot d \theta \nonumber\\ & =\mathbb{E}_{\theta\sim p_{\psi}}\!\left[\nabla_{\psi}\!\left(\log p_{\psi}(\theta)\right) \cdot F(\theta)\right] \end{align}

Equation: 30 uses the same log-likelihood technique as the popular reinforce algorithm (Sutton & Barto, Reference Sutton and Barto2018). The expectation can be computed using Monte Carlo approximations by sampling points as shown in Equation (31).

(31) \begin{equation}\begin{aligned}\nabla_{\psi}\!\left(\mathbb{E}_{\theta \sim p_{\psi}}[F(\theta)]\right) &=\mathbb{E}_{\theta\sim p_{\psi}}\!\left[\nabla_{\psi}\!\left(\log p_{\psi}(\theta)\right) \cdot F(\theta)\right] \\[3pt] & \approx \frac{1}{\lambda} \sum_{k=1}^{\lambda} \!\left[\nabla_{\psi}\!\left(\log p_{\psi}(\theta)\right) \cdot F(\theta)\right]\end{aligned}\end{equation}

Evolution Strategy for RL: Researchers in Salimans et al. (Reference Salimans2017) utilized Natural Evolution Strategy to find the optimal parameters of a neural network for RL problems. The parameters represent the weights and biases, and the objective is the stochastic return from the environment. The parameterized distribution, $p_\psi(\theta)$ is assumed to be multivariate gaussian with mean $\psi$ and a fixed noise of $\sigma^2I$ .

(32) \begin{equation}\theta \sim \mathcal{N}\!\left(\psi, \sigma^{2} I\right) \text { equivalent to } \theta=\psi+\sigma \epsilon, \epsilon \sim \mathcal{N}(0, I)\end{equation}

This parametrization allows to rewrite the average objective as $\mathbb{E}_{\theta \sim p_{\psi}} F(\theta)=\mathbb{E}_{\epsilon \sim N(0, I)} F(\theta+\sigma \epsilon)$ . Given the revised definition of the objective function, the authors derive (Salimans et al., Reference Salimans2017) the gradient of the objective function in terms of $\theta$ as

(33) \begin{equation}\nabla_{\theta} \mathbb{E}_{\epsilon \sim N(0, I)} F(\theta+\sigma \epsilon)=\frac{1}{\sigma} \mathbb{E}_{\epsilon \sim N(0, I)}\{F(\theta+\sigma \epsilon) \epsilon\}\end{equation}

The gradients are approximated by generating samples of noise $\epsilon$ during training and the parameters are then optimized using stochastic gradient ascent as shown in Algorithm 1.

Algorithm 1: Evolution strategy for RL by OpenAI (Salimans et al., Reference Salimans2017)

2.5.3. Neuroevolution

Neuroevolution is a form of Artificial Intelligence (AI) that combines EC with NNs. The objective is to either learn the topology of the NN or its parameters, that is, weights and biases by utilizing algorithms discussed in Sections 2.5.1 and 2.5.2.

An individual within the population is a NN or its parameters—weights and biases. The trivial method of storing a NN, as a complex data structure, in the parameter space scales poorly in memory, especially with the increasing population size, and makes the application of genetic operators a computationally expensive task. Hence, the need for simplified encoding of the NN.

Direct encoding: In the notable work, ’Evolving neural networks through augmenting topologies’, NEAT (Stanley & Miikkulainen, Reference Stanley and Miikkulainen2002), the authors propose a direct encoding wherein the connections between each neuron are explicitly stored with all the weights and biases, as shown in Figure 12. Such an encoding gives a very high degree of flexibility to the network topology but has a massive search space due to a fine granularity of the encoding. This approach is not well suited for DNN, which consists of several hidden layers, each with thousands of neurons.

Figure 12. A simplified version of direct encoding proposed in NEAT (Stanley & Miikkulainen, Reference Stanley and Miikkulainen2002). Mapping of genotype, a neural network to a phenotype, its encoding

Indirect encoding: This type of encoding relies on a translation function or a mechanism that maps a neural network to its encoded representation. Unlike, direct encoding which is a quick way to encode or decode a neural network, this type of encoding has a processing overhead and limited granularity. Some implementations that employ an indirect encoding include Compositional Pattern Producing Networks (CPPNs) (Stanley, Reference Stanley2007), HyperNEAT (Stanley et al., Reference Stanley, D’Ambrosio and Gauci2009) and Deep Neuroevolution (Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018).

2.6.1. Evolution-based RL

The Q-learning-based methods try to solve the Bellman optimality equation whereas, policy gradient algorithms consist of parameterizing the policy itself to learn the probability of taking an action in a state. Regardless, they usually entail expensive computations, such as gradient calculations and back-propagation, which result in very lengthy (i.e., hours to even days) training in order to obtain desirable results, especially when solving complex problems with large state spaces (Lillicrap et al., Reference Lillicrap2015; Mnih et al., Reference Mnih and Kavukcuoglu2015; Mnih et al., Reference Mnih, Badia, Balcan and Weinberger2016).

Recently we see a surge of a number of alternative approaches to solving RL problems, including the use of EC. These methods have been quite effective in discovering high-performing policies (Whiteson, Reference Whiteson2012). A notable contribution is the work by OpenAI on Evolution Strategy (ES) (Salimans et al., Reference Salimans2017) to train deep neural networks. ES does not calculate the gradients analytically, instead approximate the gradient of the reward function in the parameter space using minor random tweaks. This massively scalable algorithm, trained on several thousand workers was able to solve the 3D humanoid walking benchmark (Tassa et al., Reference Tassa, Erez and Todorov2012) within minutes and also outperformed several established deep reinforcement learning algorithms on many complex robotic control tasks and Atari 2600 benchmarks. Collaborative Evolutionary Reinforcement Learning (CERL) (Khadka et al., Reference Khadka, Majumdar, Chaudhuri and Salakhutdinov2019) is another scalable framework that was designed to simultaneously explore different regions of the search space and creating a portfolio of policies, the results on continuous control benchmarks highlighted improved sample efficiency of such approaches. Another breakthrough work by researchers at UberAI, is Deep Neuroevolution (Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018), a black-box optimization technique based on principles of evolutionary computation. Deep Neuroevolution utilizes a simple GA to train a convolutional neural network that approximates the Q-function.

Results from these publications highlight improved performance and significant speedup due to distributed training (Salimans et al., Reference Salimans2017; Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018). Moreover, these gradient-free approaches can be applied to non-differentiable domains (Cho et al., Reference Cho2014; Xu et al., Reference Xu2016; Liu et al., Reference Liu2018) and for training models that require low precision arithmetic—binary neural networks on low precision hardware. In a subsequent paper (Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018), the same researchers incorporated quality diversity (QD) (Cully et al., Reference Cully2015; Pugh et al., Reference Pugh, Soros and Stanley2016) and novelty search (NS) (Lehman & Stanley, Reference Lehman and Stanley2008) to ES, and proposed three different algorithms, which improved performance and also directed the exploration. Their work also highlighted the algorithm’s robustness against a local minima.

Several population-based evolutionary algorithms have been used to train neural networks or optimize their architecture (Jaderberg et al., Reference Jaderberg2017) and the hyperparameters (Liu et al., Reference Liu2017), but a rather interesting approach was proposed in Genetic Policy Optimization (GPO) (Gangwani & Peng, Reference Gangwani and Peng2018) algorithm which utilized imitation learning as crossover and policy gradient techniques for mutations. The algorithm had superior performance and improved sample efficiency in comparison to known policy gradient methods. Evolutionary Reinforcement Learning (ERL) (Khadka & Tumer, Reference Khadka and Tumer2018) is another hybrid approach that uses a population-based optimization strategy guided by policy gradients for challenging control problems.

3.3.1. Model encoding

An individual from the population which is a neural network, is stored as list of tuples, as shown in Figure 15.

Figure 15. Proposed encoding of a neural network

The first tuple consists of a randomly selected seed utilized for initializing the model parameters, and the second entry is the identifier for the species of the network. Every subsequent tuple consists of another seed value and the mutation power ( $\sigma$ ) utilized by the genetic operators for training the network parameters.

3.3.2. Genetic operators in encoded space

The Gaussian operator is the choice for mutation in this work. Given a neural network with parameters, $\theta$ , the Gaussian operator updates all the parameters as:

(34) \begin{equation} \theta^{\prime}=\theta+\sigma \epsilon\end{equation}

where $\epsilon \sim \mathcal{N}(0, I)$ and where $\sigma$ is the mutation power. If a parent is selected for mutation, the encoding of the resultant network is simply the original encoding appended with a new tuple containing the seed and the mutation power. This process is shown in Figure 16. When needed, any encoding can be decoded to the actual network by iteratively setting the seed and applying the Gaussian mutations.

Figure 16. Mutation of a neural network in parameter and encoded space. $\theta_{0:w}^{g-1}$ represent the parameters of the network at $g-1$ generation. $\theta_{0:w}^g$ are obtained by utilizing a gaussian operator with $\sigma_g$ as the mutation power and $\tau_g$ as the random seed

The proposed encoding is also well suited for the application of single-point crossover as described in Section 2.5.1. A random point on the encoded representation of the parents is chosen as the crossover point. A new offspring is produced by merging the left and right partitions (and vice versa) of either parent. The resulting list represents a valid encoding of a neural network. Figure 17 shows this intuitive application of crossover. This approach can also be applied to neural networks with different architectures and took inspiration from the crossover of chromosomes in genetics.

4.2.1. Model 1

Model 1 is used as the neural network for all Atari 2600 games. This includes Experiments 1, 2, 3, and 4. The architecture of the model is identical to DQN (Mnih et al., Reference Mnih, Badia, Balcan and Weinberger2016). The model takes the preprocessed training frame of size $84\times84\times4$ as the input and outputs the Q-value for all possible actions for that game. There are three layers of convolution. The first convolution layer has 32 filters of size $8\times8$ , which are applied with a stride of 4 and ReLU activation function. The second convolution layer has 64 filters of size $4\times4$ , which are applied with a stride of 2 and ReLU activation. The last convolution layer has 64 filters of size $3\times3$ , which are applied with a stride of 1 and ReLU activation. The penultimate layer is fully connected with an output of 512 neurons. The final output layer is a fully connected layer with the output neurons equivalent to the possible actions in the game. The architecture of the model is shown in Figure 18. The model consists of over 1.7M trainable parameters and has a size of about 6.75MB.

Figure 18. Neural network used for Atari 2600 games

4.2.2. Model 2

Model 2 is used as the neural network for the Remote Electrical Tilt (RET) simulator by Ericsson, that is, Experiment 5. It is a feed-forward network with four fully connected layers of size 100, 50, 20, and 3 respectively. The architecture of the model is shown in Figure 19. This model has over 7500 trainable parameters and a size of about 0.06MB.

Figure 19. Neural network used for RET environment

4.3.1. Atari 2600 games

Atari 2600 games, which are implemented in the Arcade Learning Environment (Bellemare et al., Reference Bellemare, Naddaf, Veness and Bowling2013) are available on Gym. These problems are widely used for benchmarking the performance of RL algorithms (Hasselt, Reference Hasselt2010; Mnih et al., Reference Mnih, Badia, Balcan and Weinberger2016; Wang et al. Reference Wang, Schaul, Hessel, Hasselt, Lanctot and Freitas2016; Salimans et al., Reference Salimans2017; Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018). Each game consists of a range of actions, and game scores as rewards. The state of each game is the pixel data from an image. This is also known as a game frame. Figures 21 and 22 show examples from the game frames from four Atari 2600 games.

Figure 21. Game frames from 4 Atari 2600 games—beamrider, frostbite, qbert, spaceinvader

Figure 22. Original game frame (left) and preprocessed frame (right) using the first three steps

Gym has a collection of over 50 games. Each game is implemented as an environment in the library. The agent is provided with the current state and a list of possible actions. Once an action is selected by the agent, it is given a reward (positive or negative) by the model and the next state of the environment. The interaction is repeated until the environment returns a flag indicating the termination of the game. The technical specifications of all the games utilized in this work are available in Appendix A.1.

Data Preprocessing: Most games, have an image frame of size $210\times160\times3$ pixels which puts a heavy workload when training a NN, especially with several hidden layers. To reduce the dimensionality of inputs, some basic preprocessing steps are applied as suggested by researchers in the implementation of DQN (Mnih et al., Reference Mnih and Kavukcuoglu2015).

1. 1. The game frame is first converted to a grayscale.

2. 2. The grayscale image is down sampled to the size $84\times84$ .

3. 3. The pixel values of the image are normalized between [0,1].

4. 4. A training frame of size $84\times84\times4$ is then produced by stacking 4 game frames together.

4.3.2. Remote electric tilt simulator by Ericsson

A use case of RL in telecommunications is RET optimization. For a better consumer experience and improved (), networks rely on making some adjustments to their configurations remotely. Antenna downtilt is the angle of inclination between the horizontal plan the radiating beam of the antenna which is shown in Figure 23. RET optimization is a technique to adjust the downtilt as mentioned above to improve certain Key Performance Indicators (KPIs) defined by the network. This framework is well suited for applying RL, and prior work using DQN has shown safe and reliable policies (Vannella et al., Reference Vannella2021).

Figure 23. Antennae downtilt, $\theta_{t,c}$ for cell c at time t. Source Vannella et al. (Reference Vannella2021)

The state of the environment comprises factors like congestion, overlapping, interference, overshooting and Reference Signal Receive Power (RSRP)Footnote 6. The actions include uptilt by 1 degree, downtilt by 1 degree and no tilt. After training, the simulated environment provides the overall improvement on four metrics, the Reward KPI, good traffic, bad traffic, and RRC congestionFootnote 7. The details about metrics are proprietary, however, a positive value for each metric is an indicator of a good policy.

Experiment 1: Comparison to gradient-based methods

In this experiment, the performance of Sp-GA is compared with two gradient-based RL methods, DQN (Mnih et al., Reference Mnih and Kavukcuoglu2015), a sequential algorithm and A3C (Mnih et al., Reference Mnih, Badia, Balcan and Weinberger2016), a distributed algorithm. The metrics used for evaluation are cumulative rewards and training time.

Table 2. Hyperparameters used to train Sp-GA on Atari 2600 games

Each algorithm was trained on 6 Atari 2600 games—beamrider, assault, spaceinvader, qbert, frostbite, and amidar for a maximum of 5 million training frames. The hyperparameters for DQN and A3C were obtained from the benchmarked results of rllib libraryFootnote 8 and for Sp-GA, they are mentioned in Table 2. The same architecture (Model 1) and preprocessing was utilized. The training started with a random number (maximum upto 30) of no-op action similar to prior work (Mnih et al., Reference Mnih and Kavukcuoglu2015). A single episode for DQN, A3C, and iteration for Sp-GA was capped at a maximum of 10K training frames.

Experiment 2: Comparison to gradient-free methods

In this experiment, the performance of Sp-GA is compared with two gradient-free techniques that also use EC-based methods for training, a simple GA (Conti et al., Reference Conti, Madhavan, Such, Lehman, Stanley and Clune2018), and ES (Salimans et al., Reference Salimans2017). The metrics used for evaluation are cumulative rewards.

Each algorithm was trained on 16 Atari 2600 games (list available in Appendix A.1) for a maximum of 25 million training frames. The hyperparameters for ES were obtained from the benchmarked results of rllib library. GA was implemented as a special case of Sp-GA without crossover and species-based initialization. All other parameters for GA and Sp-GA were kept the same which are mentioned in Table 2. The same architecture (Model 1) and preprocessing was utilized across algorithms. A single episode for ES and iteration for GA, Sp-GA was capped at a maximum of 10K training frames.

Experiment 3: Scalability Assessment of Sp-GA

In this experiment, the scalability of distributed Sp-GA is assessed by evaluating the speedup and efficiency. The efficacy of encoding is also tested by comparing memory utilization.

This experiment was conducted on a single Atari 2600 game—spaceinvader. Sp-GA was trained for a maximum of 500 000 training frames on clusters with different number of workers. A single iteration step was capped at a maximum of 1000 training frames. The population size was reduced to 100; all other hyperparameters were kept the same as mentioned in Table 2.

Experiment 4: Comparison of sample efficiency

In this experiment, the sample efficiency of Sp-GA, A3C, DQN, ES, and GA are compared. The algorithm that consistently obtains higher rewards with the increasing size of training frames is considered as more sample efficient. The results are computed on 3 Atari 2600 games chosen at random.

This experiment was conducted on 3 Atari 2600 games—frostbite, qbert and spaceinvader. The same training pipelines from Experiments 1 and 2 were used. The average rewards from all the algorithms were logged during training up to 5 million frames.

Experiment 5: Performance evaluation on RET optimization

The performance of Sp-GA is evaluated on the RET simulator provided by the host organization (Section 4.3.2) and compared to DQN. The evaluation metrics are provided by the simulator.

This experiment was conducted on RET simulator provided by the host organization. The implementation and hyperparameters for DQN were preconfigured and are mentioned in Appendix 3. Both algorithms were trained for a fixed number of iterations of 1500. The model architecture was kept the same (Model 2). The hyperparameters used by Sp-GA are summarized in Table 3

Table 3. Hyperparameters used to train Sp-GA on RET environment