3.1 Previous agent-based models of design

A considerable body of knowledge has been developed over time on modelling with agents. This paper will cover some of the work made on agent models specifically directed at the design process. Design is a process by which knowledge is transferred and synthesized. However, the constant evolution of standards, technologies, and a dynamic marketplace demands a high degree of adaptability in both design expertise and in the process of applying that expertise (Lander Reference Lander1997). As researchers apply multi-agent technologies to design domains, challenges include supporting interoperability among heterogeneous agents on diverse platforms, coordinating the design process, and managing conflict.

McGuire et al. (Reference McGuire, Kuokka, Weber, Tenenbaum, Gruber and Olsen1993) developed a system to enable effective information sharing and decision coordination in collaborative product development and integrated manufacturing. To that end, the SHADE (SHAred Dependency Engineering) project strikes a balance between two undesirable extremes – typical special-purpose CAE systems tend to isolate information at tool boundaries, and typical integrated CAE systems tend to limit flexibility and process innovation – by streamlining communication exchange among special-purpose CAE systems. SHADE’s approach has three main components: a shared knowledge representation (language and domain-specific vocabulary), protocols supporting information exchange for change notification and subscription, and facilitation services for content-directed routing and intelligent matching of information on consumers and producers. With this system, they hoped to support coordination among numerous heterogeneous agents. The work focused mainly on communication and translation across several platforms, and it was not aimed at the early design phase, nor at the particular problems of iteration and adaptation, although it can be argued that the problems of uncertainty and collaboration are dealt with.

Dai & Drogemuller (Reference Dai and Drogemuller1999) developed a collaborative framework specification where agent technologies can be effectively incorporated. The framework encourages modular software components interaction, accommodating various software technologies tailored to users’ requirements. They used the domain activities of compliance checking and cost analysis to demonstrate how collaborative building design may be achieved with software agents as autonomous decision-making systems. Although the system was developed specifically for building design and targets the communication across different software platforms, the aspect of collaboration is highlighted. Perhaps due to the particular setting of the problem (building design) the problems of iteration and adaptation have not been tackled.

In the area of product design there is a constant need for integration of design, manufacturability and process planning, often performed by dispersed teams in a concurrent way. Sun, Zhang & Nee (Reference Sun, Zhang and Nee2001) proposed a model considering constraints and requirements from the different product development cycles in the early development phases in order to support the concept of design for manufacturability. This model uses conflict resolution techniques and design-improvement suggestions to refine the initial product design, comprising a facilitator agent, a console agent and six service agents. Each service agent is used to model different product development phases, and the console agent acts as an interacting interface between designers and the system, while the facilitator is responsible for the decomposition and dispatch of tasks, and resolving conflicts. A prototype system was implemented and the authors concluded that it could be extended to include other service agents to become a comprehensive distributed concurrent engineering system for geographically dispersed customers and suppliers. This system tackles iteration, uncertainty and collaboration but adaptation is dealt with through a console agent. In a sense, the system is trying to mimic a context in which there is an agent that resolves conflict and distributes tasks, which means that the system cannot do it on its own just through the definition of the other agents’ behaviour.

Cao et al. (Reference Cao, Zhu, Cui and Tan2008) used an agent-based approach to guide the conceptual design phase of a mechanical product, with the aim of reducing lifecycle costs. The framework uses functional parameters and design variables extracted from the mechanical product design requirements. An agent-based framework is subsequently established with multiple agents for solving the functional requirements and proposing a design solution. The proposed model seems to handle collaboration in a sequential way, neglecting iteration, adaptation and uncertainty as a consequence.

Complex design involves trade-offs and teamwork, which implies social interaction of some kind (Crowder et al. Reference Crowder, Bracewell, Hughes, Kerr, Knott, Moss, Clegg, Hall, Wallace and Waterson2003). To make collaborative design decisions, designers must negotiate with each other by exploring the design space, generating new ideas and compromising for agreement. Jin & Lu (Reference Jin and Lu2004) built on contemporary advances in negotiation research in the area of social psychology, distributed artificial intelligence, and decision theory and applied that to design via an agent-based negotiation framework to support collaborative design decision-making. The system is particularly concerned with negotiation, which in the context of the present work translates into collaboration and adaptation. As a consequence, iteration and uncertainty are neglected in this agent model.

Multidisciplinary team working is becoming increasingly important in modern organizations, and it is characterized by complex interactions between its individual members, related to communication and dynamics between members, the working environment, and the team’s work tasks. Crowder et al. (Reference Crowder, Robinson, Hughes and Sim2012) proposed an agent-based modelling approach for simulating team work within an engineering environment. The model includes a number of variables at an individual level (competency, motivation, availability, response rate), team level (communication, shared mental models, trust), and task level (difficulty, workflow), which jointly determine team performance (quality, time to complete the task, time spent working on the task). Crowder et al. (Reference Crowder, Robinson, Hughes and Sim2012) further report the results of various simulation runs that were conducted in response to realistic team working scenarios, together with its validation, and discuss the model’s practical applications as a tool for facilitating organizational decision-making with respect to optimizing team working. The model seems to deal well with collaboration and adaptation, but not with uncertainty and iteration.

Multiple and concurrent iterations during the design phase of aircraft structural parts significantly increase the lead time and cost of new aircraft products’ development. Effective communication, collaboration and coordination of designers and software tools used by them are key mechanisms to address this challenge. Wang et al. (Reference Wang, Li, Li and Liu2012) presented an agent-based collaborative design framework to facilitate the collaboration of feature-based aircraft structural parts design and analysis tools, including detailed design, machining feature recognition, feature-based manufacturability evaluation and cost estimation. A prototype system has been implemented using JADE (Java Agent Development system) and demonstrated through a case study in collaboration with a major aircraft manufacturer in China. This work is the closest to the present work, in the sense that it tackles the four main characteristics of the early design phase. It was, however, specifically developed to solve the problem of feature-based design for aircraft structures, rendering its generalization to other problems difficult.

This section presented some previous attempts at using agents in the design process. It can be seen that these have been few and far between. They have either failed to address the common problems in the design process, especially in early design phases or lack the ability to deal with problems other than those to which they were specifically applied to. The current work will aim to address these gaps. Section 3.2 will establish a system architecture to support the model, explaining how the available tools can cater for the particularities of the early design process. Section 3.3 will define the agents’ behaviour and how they interact in the model. Section 3.4 will present how the model deals with design uncertainty and design iteration, Finally, Section 3.5 will shed some light into the modelling of collaboration and adaptation.

3.2 Architecture of the design process model

As mentioned above the motivation for developing the early design process of complex engineering systems arose from limitations of the modelling approaches reported in the literature in capturing its dynamics together with the findings of the empirical work performed at Rolls-Royce plc, a major manufacturer of jet engines, related with the lack of understanding on how change on high-level requirements, which are subject to change, impact project performance measures. This section provides an overview on the observed design organization, followed by a description of the architectural design of the agent-based model of early phase of design processes.

3.3 Agent definition, behaviours and interactions

Based on the observed behaviour and work structured of designers and design teams during the preliminary design of jet engines and using the modelling process flow sketched in Figure 2, four agent sub-types were realized in AMPERE inheriting attributes and behaviours from the generic Design Agent architecture. Replicating what was found in the design environment at the jet engines manufacturer, these four sub-types have been incorporated in the model as: a Customer agent; a project Lead agent; a Senior Designer agent; and a Junior Designer agent (Figure 3).

Figure 3. Overview of the agent definition and behaviours (agent optional tasks, interactions and communications) in AMPERE.

The Customer Agent incorporates a client entity with privileged access to the market environment: it observes how needs evolve over time, reacts to events of change and is motivated to send requests for a design proposal to a supplier organization (Figure 3). As observed during early design stages, the Customer is also prone to trigger an update of requirements when market needs have changed.

Conversely, market information is inaccessible to agents belonging to the supplier organization, but the Lead, Senior Designer and Junior Designer agents are keen to respond to requests and updates arriving from the Customer. Replicating the type of hierarchical unit found in large organizations and, particularly, observed in the jet engine manufacturer, the Lead, Senior Designer and Junior Designer agents are the basic building blocks of a design team (Figure 3). The Lead agent has the ability to dispatch directions to designer agents in the team, which are normally predisposed to accept them. The supplier organization can be composed of one or more design teams working together.

The distinction between Senior and Junior Designers captures the distinct levels of experience found in the early design projects, which results in different impact of actions performed. Senior Designers represent elements that have completed many projects and normally work during early design across multiple projects in a collaborative manner during short periods, but with the ability to strongly influence and direct the course of the design solution. Conversely, the Junior Designer represents elements far less experienced, fully committed to one or a few projects and of much lower resource cost.

Each agent was modelled in AMPERE with the ability to observe its environment and update its beliefs, and perform a finite number of actions on the environment, including interactions with other agents. Actions appear as the outcome of the practical reasoning Beliefs–Desire–Intention algorithm, which consists of updating beliefs, generating options and filtering them to identify the desired one, finding a plan that can implement the option selected and intentionally executing it.

The structured mapping of the agents’ status of beliefs to desires selected for implementation, summarized in Table 2, supports the specification of the overall behaviour of the different sub-types of agents in the model. In practice, a desire that has been selected among available options becomes an intention, i.e., an action that the agent has committed to and is expected to have some desired effect in the environment. For instance, the Customer according to its beliefs may request a design proposal from the supplier, update requirements when it is aware that the market has changed or evaluate the solution received. Depending on its awareness on the arriving of new customer requirements, project Lead is able to request a solution from the team, update arriving requirements, evaluate project risks, direct resources according to the perceived risk level and submit a solution to the Customer.

Designer Agents can either perform design activities or update the design solution to its team (Figure 3). The realization and repetition of the design tasks allows designer agents to generate or improve the status of the design solution’s quality until requirements are met, similarly to what is observed in practice during early design phases. Furthermore, Senior Designers may collaborate with Junior Designers belonging to the team, allowing a faster convergence or a better design solution, or support other projects, since they are not dedicated to a single project.

As observed during the empirical studies some of the actions that can be executed include multi-agent interactions. Multi-agent interactions were modelled relying in the exchange of communication message instances, derived from the SPADE message class indicated in Figure 2, which allow agents to find each other and pass information through the environment. Table 3 summarizes the type of messages exchanged between agents in the model using the FIPA-ACL communication language and outlines its content. For instance, communication messages are used by the Customer Agent to request a design solution to the supplier organization and simultaneously specify two key variables: the deadline for solution submission and the expected quality.

The previous definition of agents, behaviours and interactions are the fundamentals of the agent-based model of early design developed by the author with the purpose of investigating the effects of requirements change in project performance. As previously stated, the model is inspired in empirical research about the preliminary gas turbine design and aims to capture key dynamics of early design. The subsequent sections present models of the effects of uncertainty, iteration, collaboration and adaptation introduced in AMPERE to capture such dynamics.

3.4 Modelling uncertainty and iteration

Each subtype of Design Agent was built in AMPERE with a library of specialized tasks that it can select to perform. For Junior and Senior Designer Agents, planning and execution of tasks representing actual design activities enables the agents to generate a design Solution instance or operate on its status after creation. This allows designer agents to gradually improve their knowledge and thus the quality attribute of the design solution object. While the solution’s quality is lower than the required target, meaning that requirements have not yet been achieved, the agents may choose to iterate the activities. The effect of the iteration on the agent’s design Solution instance quality attribute was modelled in AMPERE according to Equation (1):

where $Q$ is the solution instance quality attribute at iteration $n$ ; $Q_{s}$ is the standard quality level the agent is able to achieve; $Q_{i}$ is the initial quality level; and $\unicode[STIX]{x1D6FC}$ is the quality progress rate coefficient. During simulation, $Q$ , $Q_{s}$ , $Q_{i}$ and $\unicode[STIX]{x1D6FC}$ take real values between [0, 1] while $n$ stores integers within the interval $[0,+\infty ]$ . When $Q$ reaches a unitary value, this represents a fully optimized design solution. Similarly, the effect of iteration on the agent’s design Task instance duration is modelled according to Equation (2):

where $D_{n}$ is the design task instance duration attribute at iteration $n$ ; $D_{s}$ is the standard duration the designer agent is capable; $D_{i}$ is the initial duration of the task; and $\unicode[STIX]{x1D6FD}$ is the duration progress rate coefficient. During simulation, $D$ , $D_{s}$ and $D_{i}$ are defined as real numbers corresponding to time units, $n$ as integers between $[0,+\infty ]$ and $\unicode[STIX]{x1D6FD}$ takes real values within $[0,1]$ .

Equations (1) and (2) thus model the effect of iteration in the solution’s quality and task’s duration through exponential learning curves (Leibowitz et al. Reference Leibowitz, Baum, Enden and Karniel2010), which is a standard way of describing that, for a given level of fidelityFootnote ¹ , further iteration improves the performance achieved by the agent, but yields increasingly lower gains (Hamade, Artail & Jaber Reference Hamade, Artail and Jaber2005).

This allows AMPERE to incorporate two empirical notions of early design iteration dynamics. The first is that design solution improves quickly in the first iterations, normally corresponding to early stages of design exploration and convergence. Then, as early design progresses to stages of refinement and repetition, an increasingly higher iteration effort is required to continue design optimization. This modelled effect of iteration is represented in Figure 4 (left). Secondly, it incorporates also the empirical notion that designers learn from experience accumulation, meaning that subsequent repetitions of activities are performed faster than initial ones. AMPERE allows also Junior and Senior Designers to have different improvement rates, according to their level of expertise.

The previous behaviours have been incorporated in AMPERE through the methods created for the Task and Solution classes. Moreover, in order to account for the effect of day-to-day variability in individual performance of agents, Equations (1) and (2) have been implemented with probability density functions associated to the standard and the initial values and to the improvement rate coefficient. Simple probability density functions, such as triangular functions, were used to capture the effects of variability during simulation.

Figure 4. Modelled effects in AMPERE of iteration according to design experience (left) and uncertainty on solution quality in the presence of a change arrival (right). Subscript $i$ denotes initial quality level and subscript $s$ denotes standard achievable quality level.

In addition to task variability, the effects of uncertainties typical of early design phases, such as events of change arising from the external environment, have also been captured in AMPERE. Figure 4 (right) illustrates that the arrival of change has been modelled through a loss of design solution quality and work efficiency that has been achieved by the agent until the event occurred. This deterioration captures the concept that change events transport designers to a state of lower knowledge, since goals have been modified and thus design space has been repositioned relative to the current solution status. The deterioration in quality and work efficiency has been modelled in AMPERE proportionally to the magnitude of change perceived by the agent.

3.5 Modelling collaboration and adaption

In addition, facets of collaborative behaviour often observed during complex early design phases have been also incorporated. One fundamental dimension of collaboration is accounting for the traditional breakdown of large pieces of work into separate and smaller parts, which are delivered by different actors. Through the agents’ Task library, AMPERE allows the definition of specific responsibilities to individual Senior or Junior Designers, such as component or discipline-related design responsibilities. For instance, one agent may be made responsible for the aerodynamic design of one component and to pass information to another agent part of the team which may be responsible for the mechanical design of the same component. Both work until they are satisfied with their individual solutions, reacting to changes that are communicated from the external environment during that process.

Since this facet of collaboration requires partial solutions to be integrated into a more global solution, the quality of this global solution (the delivered design solution for the whole system) is expected to reflect the relative importance of its parts or sub-systems. In fact, as reported by Case & Lu (Reference Case and Lu1996), the collaborative behaviour during complex design was observed at the jet engine manufacturer, including key aspects of conflict resolution, where designers work together to achieve a better overall design, i.e., towards a globally ‘optimized’ solution which is not necessarily the sum of locally optimized sub-solutions. AMPERE includes thus the Collaborative Solution class which allows agents to track and store the results from collaborative and distributed design processes. The quality of a Collaborative Solution instance is defined according to Equation (3):

where C SQ is the collaborative solution instance quality attribute; $w_{j}$ is the contribution weight of each part or discipline aspect for the global design solution quality and $Q_{j}$ is the design solution instance quality attribute of each part or aspect, being $w_{j}$ subject to the constraint:

During simulation, C SQ, $Q_{j}$ and $w_{j}$ take real values between [0, 1] and Equations (3) and (4) capture the effects of early design collaboration based in work decomposition.

Furthermore, taking advantage of the ease of implementation of concurrency in agent-based models, AMPERE allows different Designer agents to be working on alternative solutions for the same component or disciplinary aspect. This aims to capture further aspects of early design collaborative iteration, where designers typically work concurrently on several candidates for the solution in cycles of exploration and convergence (Wynn et al. Reference Wynn, Eckert and Clarkson2007; Wynn & Eckert Reference Wynn and Eckert2017). The Collaborative Solution class has been developed with methods allowing agents to screen among alternative solutions developed for the same goal and select the one that achieved the most promising quality level.

Several facets of adaptive decision-making behaviour encountered during complex early design have been also captured in AMPERE. The practical reasoning behaviour used by agents to plan their actions state is, in essence, a way of capturing in situ adaptive decision-making. This adaptation is in essence the capacity of the Lead to evaluate risks and adapt its behaviour, deciding to either direct the resources available in the design team to work on project activities or allow them to remain available to work in other projects. Risk evaluation made by the Lead Agent includes in situ estimation of the probability and impact of not meeting the Customer’s expectations.

Probability and impact are estimated real numbers between $[0,1]$ and risk is computed by the agent using the standard approach of Risk $=$ Probability $\times$ Impact. The probability; Prob, of not meeting the Customer’s expectations has been modelled in AMPERE as a function of the time available until the deadline for design proposal submission and the gap in the status of the solution quality relative to the Customer’s expected level:

where $T_{deadline}$ is the time corresponding to the deadline defined by the Customer agent and $t$ is the present time, both defined as real numbers corresponding to time units. $Q_{gap}$ , expressed as a real value between [0; 1], is the difference between the current level of solution quality achieved by the team and the level expected by the Customer agent, c1 and c2 are real constants adjusting the rate of change in the probability function.

The impact of failing to deliver has been modelled as a linear function of the gap in quality relative to the Customer’s expectations according to Equation (6):

where $m$ is a real constant adjusting the rate of change in the impact function; and Tol is a real number between [0, 1] defining a tolerance level below which the agent is no longer sensitive to the quality gap and the impact is minimum.

Based on this risk of failing to deliver on time and with the expected level of quality, the Lead Agent continuously adjusts his design team’s resources. Resources are adjusted by the Lead in a linear manner, based on the level of risk perceived at that time. This behaviour essentially models adaptive planning to the environment’s changes observed in the design organization during early design phases.

4.1 Simulation setup

The setup of this simple model for simulation was made intending to conceptualize a simple scenario: a single customer which engages with a single supplier organization comprising one design team responsible for delivering a solution to the customer Input data and results from these simulations are illustrative and provide essentially an exploratory evaluation of the potential of the agent model to capture key dynamics of early design stages, investigate the effects of expectable levels of change in project performance and support planning.

The setup essentially includes the definition of specific internal parameters adjusting the behaviour of the environment and of the design agents previously described. The environment was set up with a stochastic generation of market changes perceived by the Customer Agent with a period uniformly distributed between 10 and 20 working days, which generates events of requirements updates from the Customer to the supplier organization.

The additional setup for agents is introduced in Table 4. It shows that the Customer Agent was set up with an internal policy defining the desire to request a new solution proposal every 4 working weeks. When a request is sent, the Customer expects a response from the supplier one working day before the time to request a new proposal has arrived. Table 4 shows also that the Customer aims to get a certain number of solution proposals. After this number is reached, the Customer Agent has achieved his goal and the simulation stops. This aims to represent early design dynamics observed in Fernandes et al. (Reference Fernandes, Henriques, Silva and Moss2014), where the Customer drives the suppliers’ design process through regular requests for proposals until it makes a decision about which solution is most promising.

Within the supplier organization, a single design team composed of a Lead Agent and several Senior and Junior Designer Agents was set up for generating a design solution proposal that meets the Customer’s expectations and respond to the Customer before the specified deadline. In this simple model, the Lead Agent has an available design team composed of five designer agents: two aerodynamic Designers, one Junior and one Senior; and three mechanical Designers, where two are Junior and one is Senior. The team is capable of adapting resources and execute actions, including design activities, tackle the arrival of changes and deliver a solution proposal to the Customer.

Table 4 shows that aerodynamic designers have the ability to assess the requirements, generate a concept, compute the efficiency of the concept and evaluate the solution created. Mechanical designers are also capable of assessing new requirements and, in addition, assess updates of the aerodynamic solution generated by fellow designers in order to compute loads acting on the system. After computation, mechanical designers generate a model and compute the stress, weight and cost of their solution. Each design task was set up with specific duration parameters and an associated variability. Activity durations reported in Table 4 refer to hours of working timeFootnote ² . In addition, each design task was set up with a probability of rework equal to 10%. Triangular probability density functions were assigned to the solution quality and duration progress rate coefficients according to the agent’s level of experience (Table 4). The initial solution quality achieved by agents after the first iteration was sampled from a triangular distribution.

Simulation begins with agents registering into the SPADE platform, which allows them to be reachable and exchange messages through the platform’s communication channel. Simulation is driven by an initial request for a solution sent by the Customer to the supplier, which occurs at the simulation’s initial time step. A time-stepped approach has been implemented and the internal clock advances at constant time intervals. This clock was included in the architectural design of AMPERE and ensures time-stepping synchronization to all agents during simulation. Exploratory simulations required approximately 2500 lines of code to run, including the classes and methods previously described as the heart of AMPERE, and each simulation took 25–35 seconds to run on an Intel Core i7-3517 CPU@1.9 GHz using a 64-bit operating system.

4.2 Visualization and performance evaluation

Process visualization is a key purpose of product development process modelling since it provides support for group discussions within design organizations (Browning & Ramasesh Reference Browning and Ramasesh2007). Because of that, exploratory simulations investigated ways to picture the process output resulting from agent-based simulation to provide the generation of a design process chart as a standard post-processing feature. Figure 5 presents an annotated and detailed view over part of a complete design process chart resulting from one simulation run with the simple setup described in Table 3. The run resulted in a design process duration of 12 weeks comprising three main interaction cycles between the Customer and the supplier organization.

Several changes occurred in the market environment during the simulation, which are visible through the requirements updates sent from the Customer agent to the supplier organization. Directions arising from the Lead motivate Designers to start their design work. Figure 5 shows also the patterns of risk evaluation and adjustments in the number of resources allocated to the design process resulting from the Lead agent’s planning activities. The request for additional resources arising from the Lead’s risk evaluation triggers various Designers to engage in concurrent design iterations.

Design activity repetition patterns appear as a result of the need for iteration to improve the solution’s quality. For instance, Figure 5 reveals aerodynamic design iterations, consisting of concept generation, calculations, solution evaluation and update to other team members. Interruptions of the natural activity cycle are also depicted, as a result of agent reaction to changes arriving from the environment. Figure 5 shows in situ reassessment of requirements upon the perception of a requirements change event and the repetition of previous activities as a result of design updates arriving from other design disciplines. The design process visualized thus emerges from concurrent streams of design activities shaped by frequent social interactions and adaptive behaviours. The ability of capturing these dynamics typical of early design phases is a major strength of the agent-based modelling approach explored.

Figure 5. View of 4 weeks of the design process chart arising from a single simulation.

4.3 Evolution of internal variables

In addition to process visualization, post-processing of AMPERE simulations provides also the visualization of the evolution of internal agent variables which have an important influence in the outcome of the simulation.

The evolution of the solution quality generated by Designer Agents and perceived by the Lead during the course of one particular simulation run is presented in Figure 6. The reader should recall that solution quality is calculated using Equations (1) and (3). This evolution shows how the work performed by aerodynamic and mechanical designer agents contributes to the overall development of a collaborative solution for the system aiming to reach the level of quality expected by the Customer Agent. The collaborative solution quality level is perceived by the Lead Agent according to the weight assigned to each solution component. In this particular setup, equal importance was given to both disciplinary aspects of the design solution. Figure 6 makes also visible the effects of uncertainty in the agent’s design efforts, showing how the quality level suffered a deterioration – the ‘knock-downs’ visible around weeks 3, 6 and 9, for instance – when events of requirements change arrived from the environment.

Figure 6. Solution quality level evolution during the course of a single simulation run, as computed from Equations (1) and (3).

Figure 7. Risk perception level evolution during the course of a single simulation run.

Figure 7 presents the evolution of the level of risk perceived by the Lead agent during the course of the same simulation. The reader should recall that perception of risk, as explained in Section 3.4 is computed in terms of probability of not meeting the required solution quality times the impact of not meeting it. Probability is a function of the time available until the deadline, whereas impact is a linear function of the gap in quality relative to the expected level of quality (Equations (5) and (6)). This is an important internal decision-making variable used for resource adjustments and arises from an in situ evaluation of time left until the deadline and the current gap in quality relative to the Customer’s expectation. Analysis shows that the level of risk perceived by the Lead Agent decreases according to the rise in the solution’s quality. It is also relevant how the perceived risk rises again around week 6 following an update of requirements arriving from the Customer. The re-evaluation of risk at week 6 ultimately led the Lead to request additional resources to perform design work.

4.4 Performance evaluation

As a result of the design team resources utilization, the cost evolution during the course of the simulation time can be determined. Figure 8 shows how both the hourly and cumulative cost evolve. Peak values of hourly cost consumed by the project occurred between weeks 3 and 4 and also between weeks 6 and 7, as a consequence of time periods where all resources available in the supplier organization were performing design work in a concurrent manner (visible in Figure 5). The accumulated cost incurred by the organization during the course of 12 weeks to execute the project reached approximately 67 thousand cost units. These exploratory simulations refer to arbitrary project cost units that the reader can relate to its own monetary units.

Figure 8. Cost evolution during the course of a single simulation run. Cost values are presented in generic project cost units.

Considering a simulation run as an individual experiment with the system, performing a significant number of simulations is a standard approach to understand and characterize in a statistical manner the effects of variability in the system’s performance. Due to its importance, multi-run post-processing of AMPERE simulations were also explored.

Figure 9 depicts two measures of project performance related to the solution quality achieved by the design team and submitted to the Customer: (1) the average quality level of proposals submitted during each project; and (2) the quality level of the last proposal submitted. The cumulative frequency of the average and last proposal arise from 50 simulation runs (50 independent experiments) with the setup described in Table 4. Figure 9 compares both measures with the expected quality level that was set up for the Customer Agent and shows that expectations were exceeded in approximately 10% of the cases. Adopting a frequentist view, this enables an estimation of the probability of satisfying the Customer with the last proposal issued for a particular model setup. Based on this sample of 50 simulation runs the last proposal quality has a median of 0.824, meaning that a confidence interval can be calculated as a measure of how precisely the median based on this sample approximates the population value. The 95% confidence interval, [0.812; 0.849], shows that the median result can be trusted even for a small number of simulation runs.

Moreover, the frequency distribution of cumulative costs incurred by the projects during multi-runs provides a second dimension for project performance evaluation. Considering that organizations normally have budget constraints, this analysis allows understanding the probability of meeting or exceeding the budget available to realize early design phases. In this case, statistical treatment of 50 simulation runs resulted in a median project cost of respectively 76 thousand cost units.

Figure 9. Histogram of solution quality from 50 simulation runs.

4.5 Effects of change in performance

One of the strengths of agent-based simulation is the possibility of investigating complex cause–effect relationships in a cost-effective manner. Because of that, exploratory simulations have also addressed an investigation to the relationship between requirements change and its effects in the global project performance. The study consisted essentially in varying the external environment’s time between changes probability distribution – keeping all other setup variables constant – and observe the response behaviour. The uniform distribution defining the environment’s time between changes was varied in steps of one working week. Fifty simulation runs of the model were performed with each modified setup. The project’s performance was measured in terms of two key metrics: the solution quality achieved in the last proposal delivered by the design team within the due date; and the cumulative cost incurred by the project. Both were characterized statistically for each set of 50 simulation runs using the median of quality and cost values.

Results are presented in Figure 10. The design solution quality response behaviour reveals that there is an interval or plateau – a change in requirements triggered each 3 to 5 weeks – where variations in the time between changes in the environment have little effect in the solution quality delivered to the Customer. In addition, Figure 10 shows that further reductions in the change frequency beyond 3 weeks produces a reduction of the last proposal’s solution quality with increasing rates. On the other hand, project cost appears to behave rather linearly and inversely to the increase in the time between changes in requirements arriving from the environment. This behaviour arises from the fact that Designer Agents spent less time looking for a new design solution due to updated requirements and thus progress faster as the rate of arrival of changes decreases.

Figure 10. Effects of change in project performance. Each performance data point arises from 50 simulation runs of a particular environment setup and refers to computed median values.

Looking to the design process as a system, Figure 10 suggests that there is a stability region relative to the arrival rate of changes and an instability point, which determines the transition to a state where changes arrive faster than what the system can cope with. Such state should naturally be avoided since project performance is significantly affected. The development of more complex models of early design with a higher number of hierarchically organized design teams, each responsible for sub-system and component design and capable of communicating changes independently at different frequencies, is however required for further investigating this hypothesis.

Considering that these results arise from a very simple model, this paper argues that agent-based models such as AMPERE are a promising approach to investigate the behaviour of complex design systems and can support industrial decision-makers planning projects during early stages. One example of such planning support is the estimation of the most likely performance of projects during early design, for an expected level of arriving changes from the environment.