1 Introduction
During the last decades intelligent/autonomous software agents have been gaining more and more applications in various domains, such as power systems management (McArthur et al., Reference McArthur, Catterson and Hatziargyriou2007), flood forecasting (George et al., Reference George, Gleizes, Glize and Regis2003), business process management (Jennings et al., Reference Jennings, Faratin, Johnson, Norman, OBrien and Wiegand1996), intersection management (Dresner & Stone, Reference Dresner and Stone2008) or solving difficult optimisation problems (Lobel et al., Reference Lobel, Ozdaglar and Feijer2011) just to mention a few. The key to understand the concept of a multi-agent system (MAS) is an intelligent interaction (like coordination, cooperation or negotiation). Thus, MAS are ideally suited for representing problems that have many solving methods, involve many perspectives and/or may be solved by many entities (Wooldridge, Reference Wooldridge2009). That is why, one of major application areas of MAS is large-scale computing (Uhruski et al., Reference Uhruski, Grochowski and Schaefer2008; Bouvry et al., Reference Bouvry, González-Vélez and Kołodziej2011).
Agents play an important role in the integration of AI sub-disciplines, which is often related to the hybrid design of modern intelligent systems (Russell & Norvig, Reference Russell and Norvig2009). As the title suggests, the article deals with a hybrid evolutionary-agent approach. In most of similar applications reported in the literature (see e.g. Sarker & Ray, Reference Sarker and Ray2010; Chen et al., Reference Chen, Kambayashi and Sato2011 for a review), an evolutionary algorithm is used by an agent to aid realisation of some of its tasks, often connected with learning or reasoning, or to support coordination of some group (team) activity. In other approaches, agents constitute a management infrastructure for a distributed realisation of an evolutionary algorithm (Schaefer & Kołodziej, Reference Schaefer and Kołodziej2003).
Evolutionary processes are by nature decentralised and therefore one may imagine the incorporation of evolutionary processes into an MAS at a population level. It means that agents are able to reproduce (generate new agents), which is a kind of cooperative interaction and may die (be eliminated from the system), which is the result of competition (selection). A similar idea with limited autonomy of agents located in fixed positions on some lattice (like in a cellular model of parallel evolutionary algorithms (PEA)) was developed by, for example, Zhong et al. (Reference Zhong, Liu, Xue and Jiao2004). The key idea of the decentralised model of evolution employed by an evolutionary multi-agent system (EMAS) (Kisiel-Dorohinicki, Reference Kisiel-Dorohinicki2002) was to ensure full autonomy of agents.
Such a system consists of a relatively large number of rather simple (reactive), often homogeneous agents, which possess or produce solutions to the same problem (a common goal), it is rather closed and static, but non-determinitic (Moya & Tolk, Reference Moya and Tolk2007). Because of both computational simplicity and a huge number of the agents, the influence of each single agent’s behaviour on the overall system operation may be neglected, which allows for the efficient realisation in large-scale environments with lightweight infrastructure (Byrski et al., Reference Byrski, Debski and Kisiel-Dorohinicki2012).
The article aims to summarise research done for 15 years since the idea of EMAS was proposed by Cetnarowicz (Reference Cetnarowicz1996). First, the background information and general structure of EMAS are presented at the beginning of the paper. Next, a basic model of EMAS and several of its variations (immunological, elitist, co-evolutionary) are described along with applications in different optimisation domains (global, multi-criteria and multi-modal). Each application is illustrated with selected experimental results recalled from the authors’ publications. References show both the most important contribution of the authors and their collaborators in the field, as well as state-of-the-art that help to locate the presented work against the background of population-based metaheuristics.
2 Towards decentralised evolutionary computation
Evolutionary computation—a heuristic problem-solving approach based on models of organic evolution—has been successfully used for difficult optimisation problems for over 40 years (Fogel, Reference Fogel1998; de Jong, Reference de Jong2002; de Castro, Reference de Castro2006). This approach is particularly useful when classical computational methods turn out to be ineffective, because corresponding models are too complex, or formulas applied too complicated, or even some formulations must be rejected in the face of numerical instability of available solvers. Unfortunately, there is no guarantee to find satisfactory solutions by evolutionary algorithms, and their performance aspects for particular problems must be verified empirically. This is simply because theoretical analyses make a number of assumptions in regard to the algorithm which limit their validity in real-world scenarios. Yet, an evolutionary algorithm may often give an approximate solution with controllable adequacy, which means that a solving process can be stopped by a decision maker anytime he is satisfied.
Practice proves that an evolutionary algorithm works properly (e.g. in terms of searching for a globally optimal solution) if the population consists of fairly different individuals, that is, the so-called diversity in the population is preserved (Bäck et al., Reference Bäck, Fogel and Michalewicz1997). Yet, many algorithms tend to prematurely loose this useful diversity and therefore the risk arises that the population might get stuck in some part of the search space (e.g. in the basin of attraction of a certain local extremum instead of searching for a global one). Loosing the population diversity also limits the possibilities of their application in some areas such as multi-objective optimisation or multi-modal optimisation.
The situation described above may be related to the fact that the model of evolution employed by simple evolutionary algorithms lacks many important features observed in organic evolution (Back et al., Reference Back, Hammel and Schwefel1997). This includes dynamically changing environmental conditions, lack of global knowledge, no generational synchronisation, co-evolution of species, evolving genotype–phenotype mapping, etc. That is why many variations of classical evolutionary algorithms were proposed, introducing additional mechanisms following the most important phenomena in evolutionary biology. Among these, decomposition and co-evolutionary approaches had the most inspiring effect on the genesis of EMAS.
2.1 Decomposition and co-evolutionary techniques
Niching (or speciation) techniques (Mahfoud, Reference Mahfoud1995) are aimed at introducing useful population diversity by forming sub-populations (species). Allopatric (or geographic) speciation occurs when individuals of the same species become isolated because of geographical or social changes. Decomposition approaches of the so-called PEA model such phenomena by introducing non-global selection/mating and some spatial structure of population (Cantú-Paz, Reference Cantú-Paz1995).
In a coarse-grained PEA (also known as regional or multiple deme model), the population is divided into several sub-populations (regions, demes), selection/mating is limited to individuals inhabiting one region and a migration operator is used to move (copy) selected individuals from one region to another. In a fine-grained PEA (also called a cellular model) individuals are located in some spatial structure (e.g. grid) and selection/mating is performed in the local neighbourhood (in this case one may imagine overlapping sub-populations).
In co-evolutionary algorithms the fitness of each individual is not computed directly based on the definition of the problem to be solved, but results from interactions with other individuals in the population. In cooperative co-evolutionary algorithms, a problem is decomposed into several sub-problems solved by different algorithms in separate sub-populations (Potter & De Jong, Reference Potter and De Jong2000). Cooperation between individuals from different sub-populations takes place only during a phase of computing fitness for the complete solution. The fitness value of the individual depends only on how well the selected group composed of representatives of all sub-populations (a complete solution) solves a given problem.
In competitive co-evolutionary algorithms usually two individuals compete against each other in a tournament and their ‘competitive fitness’ corresponds to their performance in that competition (Paredis, Reference Paredis1995). In each step of the algorithm, a given individual competes with different opponents and the results obtained have an impact on its current fitness. This mechanism can be applied irrespective of the number of populations used in the algorithm—it can be used even if there is only a single population. In this case, opponents are chosen from the same population. In the case of using several populations competitors are chosen from different sub-populations. Then, individuals mate with partners coming from the same sub-population (species).
2.2 Agent-based model of evolution
In the approaches described above, the exact course of evolutionary processes depend on some context of a particular individual or sub-population (e.g. the location in the population structure). The idea of decentralised model of evolution realised as an EMAS may be considered as a step further in that direction.
Agents of EMAS represent or generate solutions for a given optimisation problem. They are located on islands (representing distributed structure of computation), which are their local environment, where direct interactions may take place. Obviously, agents are able to change their location, which allows diffusion of information and resources all over the system (Kisiel-Dorohinicki, Reference Kisiel-Dorohinicki2002).
In EMAS phenomena of inheritance and selection the main components of evolutionary processes are modelled via agent actions of death and reproduction. Inheritance is accomplished by an appropriate definition of reproduction, like in classical evolutionary algorithms. Core properties of the agent are encoded in its genotype and inherited from its parent(s) with the use of variation operators (mutation and recombination). Moreover, an agent may possess some knowledge acquired during its life, which is not inherited. Both inherited and acquired information determines the behaviour of an agent in the system (phenotype).
Assuming that no global knowledge is available (which makes it impossible to evaluate all individuals at the same time) and the autonomy of the agents (which makes reproduction be achieved asynchronously), selection is based on the non-renewable resources (Cetnarowicz et al., Reference Cetnarowicz, Kisiel-Dorohinicki and Nawarecki1996). It means that a decisive factor of the agent’s fitness is still the quality of solution it represents, but expressed by the amount of non-renewable resource it possesses. In general, the agent gains resources as a reward for ‘good’ behaviour, and looses resources as a consequence of ‘bad’ behaviour. Selection is then realised in such a way that agents with a lot of resources are more likely to reproduce, while the low level of resources increases the possibility of death. So according to classical Franklin’s and Graesser’s taxonomy agents of EMAS can be classified as Artificial Life Agents (a kind of Computational Agents) (Franklyn & Graesser, Reference Franklyn and Graesser1997).
To recapitulate, the main advantage of the approach under discussion is that it covers various specialised evolutionary techniques in one coherent model, which enables the following:
∙ Local selection allows intensive exploration of the search space and explicitly defined living space facilitates implementation in a distributed computational environment, which is similar to PEA.
∙ Evaluation of agents, or more generally, the way a phenotype (behaviour of the agent) is developed from a genotype (inherited information) depends on its interaction with the environment, like in co-evolutionary algorithms.
3 Evolutionary multi-agent system in global optimisation
Assuming that some search space U is defined and a search criterion is given by function 
$$f:U\to{\Bbb R}$$
(often called fitness function in evolutionary computation), global optimisation task is usually defined as a search for a solution:
$$\vec {x}^{{\asterisk}} {\rm =}\,arg\,\mathop {{\rm max}}\limits_{\vec {x}\in D} \,f(\vec {x}),$$
where D⊆U is a feasible set of solutions, and arg max stands for the argument of the function which gives its maximal value. For many techniques U must be a metric space with measure d defined, typically 
$$U={\Bbb R}^{n} ,\;n\in {\Bbb N}$$
. Depending on the characteristics of D set, the optimisation task may be classified as continuous, discrete, combinatorial or mixed. Constrained optimisation may also be considered (Horst & Pardalos, Reference Horst and Pardalos1995). Moreover, it is easy to see that the problem of maximisation may easily be changed into minimisation when needed, by negating the value of the function under consideration.
Solving optimisation problems with evolutionary algorithms requires choosing the encoding of solutions according to the search space definition, crossover and mutation operators appropriate for the encoding, as well as configuring a selection mechanism, or other components of specialised techniques like migration strategies for the island model of PEA (Bäck et al., Reference Bäck, Fogel and Michalewicz1997). The use of EMAS needs similar decisions concerning the encoding and variation operators, but the configuration of the system is different.
3.1 Basic model of evolutionary multi-agent system
Figure 1(a) shows the simplest possible model of an EMAS, with one type of agents and one resource (called energy) defined. Genotypes of agents represent feasible solutions to the problem. Energy is transferred between agents in the process of evaluation. When the agent finds out that one of its neighbours (e.g. randomly chosen) has lower fitness, it takes a part of its neighbour’s energy, otherwise it passes part of its own energy to the evaluated neighbour. The level of life energy triggers the following actions:
∙ Reproduction: performed when the agent’s energy raises above a certain level, followed by production of a new individual in cooperation with one of its neighbours, with genotype based on parents’ genotypes (crossed over and mutated) and part of energy also taken from its parents.
∙ Death: the agent is removed from the system when its energy falls below a certain level, the remaining energy is distributed among its neighbours.
∙ Migration: the agent may migrate when its energy rises above a certain level, then it is removed from one evolutionary island and moved to another according to predefined topology.
(a) Evolutionary and (b) immunological types of evolutionary multi-agent system
Each action is attempted randomly with a certain probability, and it is performed only when their basic preconditions are met (e.g. an agent may attempt to perform the action of reproduction, but it will reproduce only if its energy rises above certain level and it meets an appropriate neighbour).
3.2 Evolutionary multi-agent system with immunological selection
The main idea of applying immunological inspirations to speeding up the process of selection in EMAS is based on the assumption that ‘bad’ phenotypes come from ‘bad’ genotypes. Immune-inspired approaches were applied to many problems, such as classification or optimisation (e.g. Dasgupta & Nino, Reference Dasgupta and Nino2008). The most frequently used algorithms of clonal and negative selection correspond to their origin and are used in a variety of applications (Wierzchoń, Reference Wierzchoń2002).
The general structure of immunological evolutionary multi-agent system (iEMAS) is shown in Figure 1(b). A new group of agents (acting as lymphocyte T-cells) is introduced (Byrski & Kisiel-Dorohinicki, Reference Byrski and Kisiel-Dorohinicki2007). They are responsible for recognising and removing agents with genotypes similar to the genotype patterns of these lymphocytes. Another approach may introduce specific penalty applied by T-cells for recognised agents (certain amount of the agent’s energy is removed) instead of removing them from the system. Of course there must exist some predefined affinity (lymphocyte–agent matching) function which may be based, for example, on the percentage difference between corresponding genes.
Agents-lymphocytes are created in the system after the action of death. The late agent genotype is transformed into lymphocyte patterns by means of mutation operator, and the newly created lymphocyte (or group of lymphocytes) is introduced into the system. In both cases, new lymphocytes must undergo the process of negative selection. In a specific period of time, the affinity of immature lymphocyte patterns with ‘good’ agents (possessing relatively high amount of energy) is tested. If it is high (lymphocytes recognise ‘good’ agents as ‘non-self’) they are removed from the system. If affinity is low, it is assumed that they will be able to recognise ‘non-self’ individuals (‘bad’ agents) leaving agents with high energy intact. The life span of lymphocytes is controlled by specific, renewable resource (strength) used as a counter by the lymphocyte agent.
Therefore, EMAS is enhanced by adding lymphocyte agents, altering the action of the agent’s death and adding three lymphocyte-related actions:
∙ Death: EMAS action of death is redefined: during this action the agent produces one or more lymphocyte agents passing its mutated genotype to them and setting their strength to the maximum value.
∙ Killing: mature lymphocyte (with energy below a certain level) removes (or weakens) one of its neighbouring agents, if it finds that the genotype of this agent matches its own using predefined affinity function. Immature lymphocytes (with strength excessing a certain level) are checked whether they match an agent with high energy, in this case the lymphocyte is removed from the system.
∙ Apoptosis: lymphocyte with zero level of strength is removed from the system.
∙ Give: this action controls the negative selection process and overall lymphocyte acting time by simply decreasing the level of lymphocyte strength, allowing it to perform other actions (e.g. killing and apoptosis).
The concept of iEMAS is especially advantageous in applications requiring time-consuming fitness evaluation like the evolution of neural network architecture (Byrski & Kisiel-Dorohinicki, Reference Byrski and Kisiel-Dorohinicki2007).
3.3 Evolutionary multi-agent system and immunological evolutionary multi-agent system optimisation results
Experiments concerning minimisation of the benchmark functions presented below were reported by Byrski and Kisiel-Dorohinicki (Reference Byrski and Kisiel-Dorohinicki2007). EMAS, iEMAS and a classical PEA were checked against popular benchmarks (Digalakis & Margaritis, Reference Digalakis and Margaritis2002) in order to test their efficiency (10-dimensional functions of Ackley, De Jong, Griewank and Rastrigin). Variation operators of discrete crossover and uniform mutation were used. In PEA tournament selection (being the most similar selection mechanism to energetic selection principle in EMAS) was used. The systems consisted of three evolutionary islands with 30 agents (or individuals in PEA) in the initial configuration.
Best fitness values in consecutive steps of the system’s work obtained for all systems for Rastrigin benchmark are presented in Figure 2. The experiments were repeated 10 times and the average together with standard deviation was shown. It may be seen that agent-based systems outperformed the classical one (PEA) in the means of getting close to the desired solution.
Best fitness for evolutionary multi-agent system (EMAS), immunological evolutionary multi-agent system (iEMAS) and parallel evolutionary algorithms (PEA) for Rastrigin benchmark
It must also be noted that iEMAS was a little worse than EMAS, yet other advantages of this system will follow. Final results obtained for other benchmarks are presented in Table 1 (standard deviation denoted by ± sign).
Best fitness in 10 000th step of the system’s work for EMAS, iEMAS and PEA
EMAS, evolutionary multi-agent system; iEMAS, immunological evolutionary multi-agent system; PEA, parallel evolutionary algorithms
The fitness count for the presented optimisations shows that agent-based systems, EMAS and iEMAS, are far more effective than PEA, as they need far fewer computations of fitness functions to obtain similar or even better results than PEA. The efficiency results for all benchmarks are presented in Table 2.
Fitness count in 10 000th step of system’s work for EMAS, iEMAS and PEA
EMAS, evolutionary multi-agent system; iEMAS, immunological evolutionary multi-agent system; PEA, parallel evolutionary algorithms
The experiments demonstrated that though the accuracy in localising the optimum by EMAS and iEMAS in comparison to the classical PEA was similar, the efficiency was much greater (the number of fitness function calls was even one order of magnitude higher in PEA than in EMAS and iEMAS). Although EMAS’s complexity can become an important factor of the computing power consumption (cf. Wolpert & Macready, Reference Wolpert and Macready1997), this technique may still be used as effective when dealing with problems where complex fitness function is evaluated.
4 Evolutionary multi-agent system in multi-criteria optimisation
When a lot of different (often contradictory) factors have to be considered, the decision maker deals with an ambiguous situation: alternatives that are optimal from the point of view of one criterion may turn out to be absolutely unacceptable considering the others. This situation is reflected by the notion of Pareto optimality. Assuming that alternatives are evaluated by the system of M functions (objectives) denoted here by the following vector F=[f 1, f 2, … , f M]T, where 
$$f_{m} :U\to{\Bbb R},\;m=1,2,\,\,\ldots\,\,,\;M$$
(cf. Section 3), alternative 
$$\vec {x}^{a} $$
is said to be dominated by 
$\vec {x}^{b} $
(which is often denoted by 
$\vec {x}^{b} \succeq \vec {x}^{a} $
) if and only if (assuming maximisation of all the objectives):
$$\forall m \ f_{m} (\vec {x}^{a} )\leq f_{m} (\vec {x}^{b} )\,and\,\exists n\in \{ 1,2,\,\,\ldots\,\,,M\} :f_{n} (\vec {x}^{a} ) \, \lt \, f_{n} (\vec {x}^{b} )$$
The goal of solving multi-criteria optimisation problem is to find all non-dominated alternatives 
$$\vec {x}^{{\asterisk}} $$
in a feasible solution set D, which constitutes the so-called Pareto set (and in objective space the so-called Pareto frontier).
For the last 20 years a variety of evolutionary multi-criteria optimisation techniques have been proposed (see Fonseca & Fleming, Reference Fonseca and Fleming1995; Veldhuizen & Lamont, Reference Veldhuizen and Lamont2000; Deb, Reference Deb2001; Coello Coello et al., Reference Coello Coello, Lamont and Van Veldhuizen2007 for a review). Deb’s typology of evolutionary multi-objective algorithms distinguishes elitist and non-elitist algorithms for the first time (Deb, Reference Deb2001). The main difference between these two groups of techniques consists in utilising the so-called elite-preserving operators that give the best individuals (the elite of the population) an opportunity to be directly carried over to the next generation, regardless of the actual selection mechanism used. Of course, if the algorithm finds a better solution than the one in the elite, this solution becomes a new elitist solution. Unfortunately, in the case of multi-criteria optimisation, as there are multiple objective functions, it is not as straightforward as in the single-objective case to identify the elite. In such situations, the non-dominating ranking comes to our rescue (Deb, Reference Deb2001: 240).
During research to date, many different realisations of EMAS-based multi-objective optimisation problems solvers have been proposed, for example, elitist multi-objective EMAS (e.g. Siwik & Natanek, Reference Siwik and Natanek2008), EMAS with cultural and immunological mechanisms (Siwik & Dreżewski, Reference Siwik and Dreżewski2009) and the whole set of co-evolutionary multi-agent systems (CoEMASs) (Dreżewski & Siwik, Reference Dreżewski and Siwik2010). In the next section elitist evolutionary multi-agent system (elEMAS) will be briefly described as a good illustration of multi-objective EMAS.
4.1 Elitist evolutionary multi-agent system for multi-objective optimisation
Elitism can be introduced into EMAS in many different ways. In the case that is considered, it is based on a slightly modified structure of environment (see Figure 3). Compared with the structure of environment shown in Figure 1(a), modifications consist in introducing an additional, the so-called elitist island and special actions that can be performed (only) by selected agents, allowing them to migrate to this very island. The elitist island only accepts immigrant agents, there is no outbound migration from there. Thus, agents which have decided to migrate to this island are not able to go back from elitist to ordinary island(s), and cannot take part in the process of evolution.
Elitist evolutionary multi-agent system
A mechanism allowing to identify elitist agents is based on the level of additional resource called prestige, which is gathered (and is not lost) by agents during their life. At the beginning of their life, the level of prestige equals zero. Then, every time an agent dominates (in a sense of domination relation) any other agent, its level of prestige increases, and so it may be assumed that agents with a high level of this specific resource belong to the elite. This mechanism allows the realisation of the above-mentioned idea of non-dominating ranking in a really elegant, easy to understand and implement way that does not require any additional complicated operations and computations.
It turns out that applying elitist operator(s) to an EMAS—despite other presented advantages—gives an opportunity to introduce additional mechanism(s) responsible for improving the spreading of the individuals over the Pareto frontier. One of such mechanisms can be realised as follows: during meetings, an agent is able to check if a solution represented by its opponent is located in its surroundings at the frontier (i.e. if the distance between solutions represented by agents is less than a given value ε). If so, the agent increases its personal counter storing the number of individuals located in the same fragment of the Pareto frontier. In addition, asking its opponents about the number of individuals located in their surroundings, the agent is able to gather with time (partial) knowledge of the average number of individuals located in some areas of the Pareto frontier (i.e. in surrounding areas represented by agents it met). Consequently, if the agent becomes the elitist one, it can make a decision about migrating to the elitist island only when the number of individuals located in its surroundings is greater than the average number of individuals located in other regions of the Pareto frontier.
The mechanism described allows deeper exploration of those areas of the Pareto frontier which are worse sampled than its other regions. A similar mechanism can also be applied to the space of decision variables, thus concerning the Pareto set. Of course both mechanisms can be used simultaneously, and this can be of great importance in real-life problems where both, Pareto set and Pareto frontier, are often disconnected.
In addition, ε parameter can be managed adaptively by agents. In the simplest form, it can be linearly smaller and smaller with the agent’s lifetime. In more advanced implementations it can be decreased by agents on the basis of their interactions with the environment and other agents.
4.2 Multi-criteria optimisation in noisy environment
Considering advantages of EMAS-based optimisation solvers over classical evolutionary algorithms, the so-called soft selection has to be underlined. Unlike classical algorithms, if an individual is dominated by another one, it is not immediately removed from the population but its energy level is decreased (depending on parameter values) so it has to be dominated several times to be eventually removed from the population.
Such a mechanism can be especially important if optimisation in a noisy environment is considered. Unlike mathematical models, in real-life problems noise cannot be avoided and it may come from many different sources, for example, distortions in input data, wrong sampling, as well as approximation faults. In such an environment, worse solutions may be falsely perceived by the algorithm as better ones. Consequently, worse solutions may be carried over to the next generations. It depends of course on the ‘scale of noise’ but it is possible that in this case the algorithm will drift towards the misleading ‘optimal’ result set.
Considering noise at the level of fitness function evaluation, this noisy fitness function takes the form of F noise=F+ξ, where F is the fitness function without noise influence and ξ the function of noise (Bui et al., Reference Bui, Essam, Abbas and Green2004). To simulate noise, normal or uniform distribution can be used. Noise used for experiments was generated with Gaussian distribution N(μ, σ), where μ=0 and σ varies from 0 to 0.20 (which represents different levels of noise). As the magnitude of objective functions for the Pareto frontier is between [0, 1], the amount of noise (especially for value 0.20) is significant. Such a range of noise allows to present the influence of noise on the quality of the result sets obtained, and consequently present the quality of algorithm.
In order to compare the performance of Non-Dominated Sorting Genetic Algorithm 2 (NSGA2) (as a commonly used benchmark) and elEMAS, all common parameters are set equally, especially population size and parameters of variation operators. Values of the most important ones are presented in Table 3. As hypervolume or hypervolume ratio (HVR) (Van Veldhuizen, Reference Van Veldhuizen1999) allows to estimate both the convergence to as well as distribution of solutions over the whole approximation of the Pareto frontier, it is one of the most commonly and most frequently used measure as the main metric for comparing quality of the result sets obtained.
Common parameters for Non-Dominated Sorting Genetic Algorithm 2 and elitist evolutionary multi-agent system
Owing to space limitations, only some exemplary but demonstrative results are presented here. As can be seen on the diagrams in Figure 4 showing values of HVR metric for Zitzler ZDT1 problem (Zitzler, Reference Zitzler1999), although NSGA2 algorithm is commonly perceived as one of the most effective and most commonly used evolutionary algorithm for multi-objective optimisation, however, elEMAS seems to be certainly a better alternative in a noisy environment. What is more, the higher noise is, the worse the results obtained by NSGA2 are. Such a characteristic is not observed in the case of EMAS-based algorithms. So it can be said that a ‘soft’ selection mechanism allows to avoid mistakes resulting from noise. For a relatively large amount of noise, elEMAS unlike NSGA2 manages to converge to the Pareto frontier.
Comparison of HVR(t) values with different values of σ obtained for solving ZDT1 problem. (a) σ=0.01; (b) σ=0.05; (c) σ=0.10; σ=0.20. HVR, hypervolume ratio; NSGA2, Non-Dominated Sorting Genetic Algorithm 2; elEMAS, elitist evolutionary multi-agent system
5 Evolutionary multi-agent system in multi-modal optimisation
The goal of multi-modal optimisation is to find all local maxima of the f(⋅) function in a feasible solution set D (cf. Section 3). It means that all points 
$$\vec {x}^{{\asterisk}} \in D$$
are sought that the following condition is true (Bäck & Schwefel, Reference Bäck and Schwefel1996):
$$\exists {\varepsilon}\,\gt\, 0\quad \forall \vec {x}\in D\quad d(\vec {x},\vec {x}^{{\asterisk}} )\,\lt\, {\varepsilon}\Rightarrowf(\vec {x})\,\leq \,f(\vec {x}^{{\asterisk}} )$$
The goal of evolutionary multi-modal optimisation techniques (niching techniques) is to find all (or most) of these solutions (Mahfoud, Reference Mahfoud1995).
In classical evolutionary algorithms, forming and stable maintaining sub-populations (species) are achieved by applying niching techniques like allopatric speciation based on the restricted application of selection and/or recombination mechanisms, as described in Section 2. Alternatively, in sympatric speciation new species evolve while inhabiting the same geographic region. It is modelled in evolutionary algorithms by techniques based on the modification of the parent selection mechanism like fitness sharing (FS), or techniques based on the modification of the mechanism of selecting individuals for a new generation like deterministic crowding (DC) (Mahfoud, Reference Mahfoud1992). Ideally, each of the species should be placed within one of the basins of attraction of local maxima, which in practice might be difficult to achieve.
Although co-evolutionary techniques and sexual selection mechanism (Sánchez-Velazco & Bullinaria, Reference Sánchez-Velazco and Bullinaria2003) promote the population diversity, they were not widely used as a basis for constructing niching techniques. These techniques were a point of reference for the model of CoEMAS (Dreżewski, Reference Dreżewski2003).
5.1 Co-evolutionary multi-agent systems
As in the case of classical evolutionary algorithms, basic EMAS-based systems, without any special mechanisms that promote useful population diversity are unable to solve multi-modal optimisation problems because they are not able to form species located within basins of attraction of many different local optima. They would rather locate only one solution, however, population diversity would be much higher than in the case of classical evolutionary algorithm.
The basic idea behind the CoEMAS model is to introduce the possibility of co-evolution of species and co-evolution of sexes (sexual selection) within EMAS. In the exemplary systems presented in the next sections, co-evolutionary interactions between species and sexual selection are used in order to construct niching techniques in EMAS and to promote useful population diversity.
In the co-evolutionary multi-agent system with co-evolving species (nCoEMAS) there exist two different types of computational agents. They will be called ‘niches’ and ‘individuals’ (agent n and agent i in Figure 5(a)). Individuals ‘live’ within niches; they can search for a suitable agent-niche and migrate into it. They can even ‘create’ new agent-niche when they are not able to find a suitable niche and migrate into it. The main goal of the mechanism of migrating between and creating new niches is to split the population of agents-individuals into such sub-populations (species) so that each of them would be located within a distinct basin of attraction of local maxima.
Co-evolutionary multi-agent systems (CoEMAS). (a) CoEMAS with co-evolving species. (b) CoEMAS with sexual selection
Mating is restricted only to agents-individuals located within the same agent-niche. Reproduction, as in the case of basic EMAS model, is initiated by an agent that has enough resources to reproduce. The agent initiating a reproduction process searches for other ready for reproduction partners from the same niche and chooses one of them. Two children are generated and each offspring receives some amount of resources from each of their parents.
Agents-niches can migrate within an environment, but only when they have enough resources, because each migration costs some amount of resources, which is returned to the environment. Two agents-niches can merge with each other: merging takes place when centres of gravity of sub-populations, which belong to two different niches, are located within the basin of attraction of the same local optima. The system is applied to multi-modal optimisation problems, therefore, the goal is to localise most of basins of attraction of local maxima.
Resources circulate between an environment and agents. The whole amount of resources within the system is constant so the number of agents cannot grow infinitely. Environment gives equal amount of resources to each agent-niche, and then they redistribute resources among agents-individuals proportionally to their fitness.
Co-evolutionary multi-agent system with sexual selection (sCoEMAS) utilises co-evolution of sexes and sexual selection in order to obtain the effect of speciation. In sCoEMAS there exist two sexes: female and male (agent X and agent Y in Figure 5(b)).
Reproduction process is initiated by a female agent when it has enough resources to reproduce. Generally, female agents use more resources than male agents during the reproduction process and that is why they reproduce less frequently than male agents. Therefore, there are always more male agents ready for reproduction than female agents and female agents choose partners from several ready for reproduction male agents. A female agent located within an evolutionary island searches for a partner in such a way that it chooses one ready for reproduction male agent from the same island. The partner is chosen on the basis of genotype similarity. The more similar two agents from opposite sex are, the more probable is that the female agent will choose the male agent. Reproduction takes place when a pair composed of agents of the opposite sex is formed. Such a pair of agents is formed permanently for several steps of the algorithm because of difficulties and costs involved in searching for a partner. The offspring (two children) is generated with the use of mutation and recombination operators and receives some of the resources from parents: more from the female agent and less from the male one.
Agents can migrate within an environment but it costs some amount of resources. Similar to nCoEMAS, in sCoEMAS the whole amount of resources present within it is constant and the environment distributes resources between agents proportionally to their fitness.
5.2 Multi-modal optimisation results
In this section, selected results of experiments with nCoEMAS and sCoEMAS applied for function minimisation are presented (the results were reported in Dreżewski, Reference Dreżewski2006 and Dreżewski & Cetnarowicz, Reference Dreżewski and Cetnarowicz2007). The results of co-evolutionary systems are compared with those obtained by classical niching techniques: DC and FS mentioned earlier and the results obtained by the basic EMAS. During experiments, Rastrigin and Schwefel multi-modal test functions were used.
The results presented in Figures 6 and 7 show the location of individuals within the search space during typical experiments with nCoEMAS and sCoEMAS and Schwefel test function. It can be seen that nCoEMAS forms very focused species, and individuals are located within the closest neighbourhood of local minima. The population diversity within species is not very high. Species are stable: once formed they do not disappear during the experiment. In the case of sCoEMAS, they are also formed and stably maintained during the experiment. However, diversity within species is much higher: agents are dispersed over the whole basins of attraction of local minima.
Species formation processes in co-evolutionary multi-agent system with co-evolving species during a typical experiment with Schwefel test function: (a) step=0; (b) step=5000
Species formation processes in co-evolutionary multi-agent system with sexual selection during a typical experiment with Schwefel test function: (a) step=0; (b) step=5000
Figure 8 shows the average number (from 20 experiments) of local minima neighbourhoods located. The local minima neighbourhood was classified as ‘located’ when there was at least ni min=3 individuals closer than 0.05 (in the case of Rastrigin function) and 10.0 (in the case of Schwefel function) to these local minima.
Average number of located minima of test functions by deterministic crowding (DC), fitness sharing (FS), evolutionary multi-agent system (EMAS), co-evolutionary multi-agent system with co-evolving species (nCoEMAS) and co-evolutionary multi-agent system with sexual selection (sCoEMAS): (a) Rastrigin; (b) Schwefel
nCoEMAS obtained the best results: on average, it located about 22 local minima neighbourhoods of Rastrigin function, and over 40 local minima neighbourhoods of Schwefel function. DC algorithm quickly located about 13 local minima neighbourhoods of Rastrigin function and about 36 local minima neighbourhoods of Schwefel function, but there was quite a strong tendency to lose almost all of them during the rest part of experiments. Species were not stably maintained in this case. FS technique detected and stably maintained about 14 local minima neighbourhoods of Rastrigin function and about 20 local minima neighbourhoods of Schwefel function on average. sCoEMAS located about six local minima neighbourhoods of Rastrigin function and five local minima neighbourhoods of Schwefel function. The results point to high population diversity. Agents are spread over the whole basins of attraction and they are not concentrated in the closest neighbourhoods of local minima. EMAS without any niching mechanism was not able to stably populate more than one local minima neighbourhood. However, what is not shown in Figure 8, it maintained high population diversity, but the whole population was located within the basin of attraction of only one minimum.
6 Conclusion
The paper presented the development of agent-based evolutionary computation paradigm realised as an EMAS. For over 15 years since it was proposed for the first time, the paradigm itself as well as its extensions and modifications has proved to be an efficient tool in different applications in global, multi-modal and multi-criteria optimisation. Various extensions, such as immunological, elitist and co-evolutionary ones have been proposed and described. Over 80 papers, four PhD and many MSc dissertations have been prepared and published so far as well. A large-scale infrastructure consisting of several clusters and a cloud-oriented virtualisation server has been prepared and is utilised in computing experiments.
Owing to the space limit, many aspects of EMAS could not be discussed in details here. This includes self-sustainability that may be applied to maintaining stability of the system (with respect to an appropriate size of the population of agents), the capability to respond to the emergency situations occurring in the hardware (breakdown of computing nodes), as well as in the user requirements (e.g. change of computation parameters) (Byrski et al., Reference Byrski, Kisiel-Dorohinicki and Carvalho2010). EMAS was also applied as a base for hybrid soft computing solvers for classification and prediction problems, with neural networks (Byrski & Kisiel-Dorohinicki, Reference Byrski and Kisiel-Dorohinicki2007), fuzzy systems (Fusinska et al., Reference Fusinska, Kisiel-Dorohinicki and Nawarecki2007), pattern recognition (Byrski et al., Reference Byrski, Dobrowolski and Toboła2008) and others.
EMAS-based approaches inevitably shares some of the flaws and limitations of evolutionary algorithms (cf. Section 2), including difficulties in their configuration and formal modelling. Some approaches developed in theoretical analysis of evolutionary algorithms (see, e.g., works of Vose or Rudolph) have been applied to EMAS—several formal models have been designed, aimed at describing the paradigm and analysing of its asymptotic features (see e.g. Schaefer et al., Reference Schaefer, Byrski and Smołka2009). Unfortunately, they take into consideration only particular cases of algorithms. As in the case of traditional (non-agent based) evolutionary algorithms as there are no accurate, general models, formal analysis of their behaviour or conclusions about their effective configurations are really limited.
Research on EMAS continues, and its main directions that should be mentioned here are the continuation of formal modelling focused on researching different asymptotic features, as well as building infrastructure and tools supporting the implementation of various systems. Applying EMAS for solving optimisation problems in many different areas with varying features and characteristics makes it necessary to utilise flexible configuration mechanisms, which drives further development of distributed software frameworks, that will allow the construction of agent-based computing systems consisting of reusable components (Pietak et al., Reference Pietak, Wós, Byrski and Kisiel-Dorohinicki2009).
Acknowledgement
The work presented in the paper was partially supported by Polish National Science Centre research project no. N N516 500039 “Biologically inspired mechanisms in planning and management of dynamic environments” and AGH University of Science and Technology statutory fund.
