1. Introduction
Designers frequently face decisions requiring the evaluation of trade-offs, often relying on a subset of available data or known alternatives to identify optimal solutions under time and resource constraints. Consequently, rational decision-making under uncertainty becomes crucial, particularly in the later stages of design, to efficiently allocate resources and minimize costs associated with unforeseen design changes while preserving the overall quality of the engineering system.
From Rationality to Bounded Rationality. Several models have been proposed in the literature to explain human decision-making behaviors under uncertainty. One prominent example is the von Neumann–Morgenstern utility theory, which defines an individual’s utility as the sum of the values of mutually exclusive decision alternatives, each weighted by their respective probabilities (Reference Von Neumann and MorgensternVon Neumann and Morgenstern, 2007). According to this theory, a rational choice maximizes expected utility. However, designers are known to exhibit bounded rationality in practice (Reference Gurnani and LewisGurnani and Lewis, 2008). Bounded rationality describes the constraints on human judgment due to limited cognitive abilities and restricted capacity to process information (Reference Simon, Eatwell, Milgate and NewmanSimon, 1990). It suggests that individuals typically adopt a satisficing strategy—seeking solutions that adequately meet their objectives rather than exhaustively searching for optimal solutions. For instance, Taylor Reference Taylor(1975) noted that when decision-makers encounter information overload, they simplify problems and disregard certain options, further constraining their rationality.
Alternative frameworks, such as prospect theory (Reference Kahneman and TverskyKahneman and Tversky, 1979) and regret theory (Reference Loomes and SugdenLoomes and Sugden, 1982), help to explain why individuals deviate from complete rationality. These theories demonstrate that people employ mental shortcuts such as emphasizing relative gains or losses rather than absolute outcomes to simplify decision-making. Thus, examining how individuals manage cognitive limitations, and psychological factors can help researchers better understand and predict inconsistencies in human decisions (Jordao et al., Reference Jordao, Costa, Dias, Pereira and Santos2020; Hernandez and Ortega, Reference Hernandez and Ortega2019; Camerer, Reference Camerer1998).
Influence of Competitive Pressures and Dynamic Decision-Making. Bounded rationality arises not only from cognitive limitations but also from external factors, such as complex decision environments and competitive pressures. In competitive environments, decision-makers often experience increased cognitive strain, shifting from pure optimization toward strategies involving overcompensation—such as allocating additional resources to match or surpass competitors’ perceived performance. Chan and Ybarra Reference Chan and Ybarra(2002) demonstrated that individuals tend to underestimate the capabilities of their partners but overestimate those of their opponents during competition. Similar competitive behaviors are observed in engineering design contexts; for instance, our previous study indicated that designers often overestimate their opponents’ performance, resulting in excessive resource allocation or overly cautious design choices aimed at mitigating perceived risks (Reference Bayrak and ShaBayrak and Sha, 2021).
Traditional game theory provides a foundational framework for understanding how decisions are optimized based on expectations of competitors’ actions. Research on dynamic competitive decision-making further shows that individuals frequently adjust strategies in response to competitor actions, continuously updating beliefs—a process modeled as a dynamic game (Reference Chen and WangChen and Wang, 2010). This adaptive behavior compels designers to iteratively refine strategies, especially under uncertainty or limited information. Consequently, competitive behavior demonstrates considerable flexibility, with decision-makers regularly adapting choices based on cognitive assessments and external pressures (Reference Tolston, Finomore, Funke, Mancuso, Brown, Menke, Riley, Hale and StanneyTolston et al., 2017).
Sequential Decision-Making and Bayesian Inference in Engineering Design. In engineering design, designers typically engage in sequential decision-making due to the cognitive load associated with considering numerous solutions simultaneously and the need to adapt to dynamic conditions, such as competitor strategies or market fluctuations. Consequently, they refine their strategies iteratively across multiple stages. One critical aspect of rational decision-making in this context is forming and updating beliefs about uncertain factors, such as user preferences, technological advancements, or competitor actions. This process resembles Bayesian inference, wherein prior beliefs are updated based on new information. Engineers start with prior knowledge from past projects, market trends, or simulations, and as new data emerges—through prototyping, competitor releases, or experimental findings—they update these priors, refining their expectations and adjusting design decisions accordingly.
Our Contribution to Designer Rationality Assessment. The existing literature extensively describes human factors contributing to bounded rationality. However, there is a gap for quantitative metrics assessing designer rationality throughout the design process, especially in competitive environments. In this paper, we propose a mathematical approach inspired by both the von Neumann–Morgenstern utility principle, where rational decisions maximize expected utility. We develop a rational design agent that performs exploration within an unknown design space based on Bayesian inference, serving as an ideal baseline for quantifying the rationality of human decision-makers. We use the similarity between human design exploration and Bayesian inference to illustrate how human subjects deviate from rational behavior. We present results from our rationality assessment using experimental data from a study in which designers explored an unknown design space under competitive conditions.
2. Experiment Setup
The analysis in this study is based on the design decision-making data collected from a prior experiment designed to examine decision-making in 1 vs. 1 design competition (Reference Sha, Kannan and PanchalSha et al., 2015; Reference Panchal, Sha and KannanPanchal et al., 2017). Participants of this experiment played a function optimization game against an opponent. Each participant’s objective was to sample values within a design space to minimize an unknown function F(x) while incurring a sampling cost per trial. The participant who found a lower F(x) wins specified number of tokens Π, minus any sampling costs. Participants were paired with different opponents in each round to eliminate reputation effects. Four sessions were conducted in total. Each session had multiple participants, each completing 15 rounds per treatment. To account for potential learning effects, the first five rounds were excluded from the analysis. The unknown function had a quadratic form with constants a and b, randomly chosen in each period within specified boundaries, given by F(x) = (x − a)2 + b, where x ∈ [−100,100] and (a, b) ∈ [−70,70]. Each trial allowed participants to sample a value of x, and the corresponding F(x) value was revealed to them at a cost c for each sample. The experiment involved two cost treatments: a low-cost treatment (c = 10 tokens per trial) and a high-cost treatment (c = 20 tokens per trial), with a fixed winning prize of Π = 200 tokens for both cost treatments.
This function optimization game represents an abstraction of a configuration design task or a parametric design problem in the detailed design stage where a design domain has been well defined with decision variables, and the artifact being designed is constructed from a predefined set of components (McComb et al., Reference McComb, Cagan and Kotovsky2017; Baldwin and Clark, Reference Baldwin and Clark2000). Sha et al. Reference Sha, Kannan and Panchal(2015) states that the nature of this game is sufficiently general to capture some of the key characteristics of engineering design processes, as outlined below.
-
(C1) “A designer’s goal is to find the best design quantified by certain objective values.”
-
(C2) “Designers assess the performance of potential designs using simulations or physical experiments.”
-
(C3) “The design search process incurs costs, which may include monetary expenses or resource expenditures such as computational power and human effort.”
-
(C4) “Investing more resources in exploring the design space enhances understanding, leading to higher-quality designs.”
The context-free configuration of this game helps reduce variations among decision-makers due to differences in domain knowledge, thereby minimizing noise in the data. The experimental setup induces two types of uncertainty: uncertainty in the design space due to the unknown objective function and uncertainty in opponent performance under competition. This setup requires participants to sequentially make two essential decisions under uncertainty in each period of the game: the choice of the next x and the decision of whether to stop or continue.
3. Methodology
We follow (Reference IcardIcard, 2018) and use Bayesian Optimization (BO) (Reference FrazierFrazier, 2018) to model a rational decision-making process under design space uncertainty, as BO—where the Gaussian Process (GP) posterior is updated after each function evaluation—appears to align with the strategy humans might naturally use (Reference Borji, Itti, Burges, Bottou, Welling, Ghahramani and WeinbergerBorji and Itti, 2013). Furthermore, GP models effectively describe how humans learn functions, particularly for simple functional relationships like linear and quadratic patterns (Reference Griffiths, Lucas, Williams, Kalish, Koller, Schuurmans, Bengio and BottouGriffiths et al., 2008). BO with different acquisition functions has been used to model and mimic human decisions in the literature (Reference Chaudhari, Bilionis and PanchalChaudhari et al., 2020). In our study, BO does not mimic human decision-making but rather serves as a baseline model to assess the rationality of the participating designers in the experimental data using distance measures. In this section, we provide a brief introduction to the preliminaries of BO and how it is implemented with the experiment data in the reference study to assess design rationality.
3.1. Bayesian Optimization
BO is a probabilistic model-based optimization method commonly used to find the global optimum of expensive-to-evaluate functions. A typical BO process comprises two major components (i) A statistical inference method, typically Gaussian process regression, serves as a surrogate model to approximate the underlying function; (ii) an acquisition function to decide where to sample in the design space which will be the core component to model a rational search process in this study.
Gaussian Process. The goal of Gaussian processes (GP) is to learn the underlying distribution of the objective function F(x) from a set of observations
${\mathscr D} = ({\bf{x}}_{1:k} ,{\bf{f}}(x_1 :k))$
(Reference Rasmussen, Bousquet, von Luxburg and RätschRasmussen, 2004). GPs define a distribution over functions, assuming that the surrogate model f(x) is jointly Gaussian and any finite set of dataset
${\mathscr D}$
follows a multivariate Gaussian distribution as in Eq. 1.
$${\rm{f}}({{\rm{x}}_{1:k}}) \sim G \mathcal P({\mu _0}({{\rm{x}}_{1:k}}),\sum\nolimits_0 {({{\rm{x}}_{1:k}},{{\rm{x}}_{1:k}}))} $$
where the mean vector, μ
0(x
1:k
) = [μ
0(x
1),…, μ
0(xk
)] evaluated by each x
1,…, xk
, and, the covariance matrix, Σ0(x
1:k
, X
1:k
) is constructed by the covariance Σ0(.,u.) between each observation. Given the observation data
${\cal D}$
, the posterior probability distribution is defined as Eq. 2.
where the posterior mean μ(x) and variance σ 2(x) represent the prediction of the model and the corresponding uncertainty, respectively, in the objective function at the point x. We set the prior mean μ 0 to zero and employ the Matérn kernel to construct the covariance matrix.
Acquisition function. BO decides where to sample next by minimizing or maximizing its acquisition function. Building upon the surrogate model, acquisition function allows searching the design space strategically in regions where the uncertainty is significant (exploration) and where the model predicts high values (exploitation). Several acquisition functions are widely used in BO, such as Expected Improvement (Jones et al., Reference Jones, Schonlau and Welch1998), and Lower Confidence Bound (Srinivas et al., Reference Srinivas, Krause, Kakade and Seeger2012). Following the von Neumann–Morgenstern utility theorem, we use the Expected Improvement (EI) as the acquisition function to model a rational search process. EI aims to strike a balance between exploration and exploitation as follows:
$$\alpha _{{\text{EI}} (x;\xi ) = (\mu (x) - f(x^*) - \xi )\Phi \left({\mu (x) - f(x^*) - \xi \over \sigma (x)}\right) }+ \sigma (x)\varphi \left({{{\mu (x) - f(x^*) - \xi }} \over {{\sigma (x)}}}\right)$$
where x* is the current best value, φ and φ are the cumulative distribution and probability density functions, respectively. A hyper-parameter ξ is used to control the exploration-exploitation behaviors.
3.2 Rationality Assessment
In this paper, we define a rational decision-making process as choosing the highest expected utility under uncertainty, as per Von Neumann-Morgenstern Utility Theorem. In black-box design space exploration, a BO process with EI as the acquisition function meets the requirements of Von Neumann- Morgenstern rationality. We quantify the rationality of participants as a continuous measure using the absolute distance between a participants guesses and the points evaluated by EI using the past decisions.
Let
${\cal D}_n^{(S)} = \{ (x_1^{(S)} ,F(x_1^{(S)} )),(x_2^{(S)} ,F(x_2^{(S)} )), \ldots ,(x_n^{(S)} ,F(x_n^{(S)} ))\} $
be the set of the first n sampled points by participant S. We train a GP model on
${\cal D}_n^{(S)} $
, constructing a surrogate function
$f(x)|{\cal D}_n^{(S)} \ {\sim}\ {\cal G}{\cal P}(\mu _n (x),\sigma _n^2 (x))$
. The total variance of the GP estimation is computed as the discrete sum
$\mathop \sum \limits_{k \in {\cal K}} \sigma _n^2 (x_k )$
over a grid
${\cal K}$
in the design space, which serves as a measure of uncertainty. Then, BO employs EI to obtain an information acquisition function
$\alpha _{{\text{EI}}} (x|{\cal D}_n^{(S)} )$
to determine the next sampling location:
where the corresponding objective function value
$F(x_{n + 1}^{({\text{EI}})} )$
is recorded for comparison and is denoted as
$y_{n + 1}^{({\text{EI}})} $
for brevity. The BO process proceeds iteratively from the (n + 1)th decision to the final decision, updating the dataset to retrain the GP at each step n:
where e represents the total number of decisions made in each trial which is different for each S. The set of decisions predicted by EI,
$\{ (x_i^{({\text{EI}})} ,y_i^{({\text{EI}})} )\} _{i = n + 1}^e $
, is compared with the actual decisions made by participant S,
$\{ (x_i^{(S)} ,y_i^{(S)} )\} _{i = n + 1}^e $
, using absolute distance measures in both the design and objective spaces. Finally, we compute the mean absolute difference between the decisions predicted by EI and the actual decisions made by participant S over i, as described in Eq. 6. Additionally, we calculate the mean total uncertainty that each S represents in each round, as shown in Eq. 7.
The absolute difference between EI and S represents instantaneous rationality of S at iteration i in the search process, and
$\mathop \rho \nolimits^{\!\!\! -}_{y} $
and
$\mathop \rho \nolimits^{\!\!\! -}_{y}$
represent the overall rationality as an average quantity. We choose absolute distance as the simplest measure for 1-D design space exploration while for higher dimensions, other distance measures can be considered. To test the relationship between rationality and uncertainty, we measure design uncertainty Ū, using the mean of the total variance in the GP estimation at each n. Also, to test whether rationality
$\bar \rho $
is influenced by the distance to the optimum function value, we define a proximity measure,
$\bar \delta $
, to the true optimum as a performance measure of the participant S.
$$\bar \delta _x = {\frac{1}\over{{e - n}}}\mathop \sum \limits_{i = n + 1}^e |x_i^{(S)} - x^*|,\bar \delta _y = {\frac{1}\over{{e - n}}}\mathop \sum \limits_{i = n + 1}^e |y_i^{(S)} - f(x^*)|. $$
where x* represents the location of the minimum function value f(x*).
4. Results and Discussion
The game dataset used in this study comprises 3,224 individual decisions. Calculating the mean performances reduces the size to 757 aggregated data points. To analyze this dataset, we set the exploration-exploitation parameter (ξ) in Eq. 3 to 0.01 and use a length scale of l = 1 and a smoothness parameter of v = 1.5 in the GP. The number of design iterations performed by the participants was limited due to cost constraints in the experiment. Thus, we start with n = 3 to train the GP and start the analysis with each participant’s fourth decision onward. We omit the data points where participants finished the round with fewer than four design samples.
4.1. Nature of the Data and Methods of Analyses
An initial analysis on the measures described in Sec. 3 show that the original values in the raw data do not follow a normal distribution. As seen in Fig. 1(a), the data is highly skewed and contains many outliers.
Comparison of the Data Before and After the Transformation
Since our primary objective is to examine the Pearson correlation between the metrics introduced in Sec. 3.2, approaching normality is essential for ensuring the validity of the correlation coefficients and for reducing skewness and kurtosis, thereby enhancing the confidence in our analyses.
4.1.1. Data Transformation
The raw data in Fig. 1(a) shows that all variables demonstrate significant right-skewness; most data points are closer to the lower end of their ranges, with a long tail extending to higher values. To address this skewness and improve normality, we first shift the data to ensure all values are positive, anchoring the minimum values at 1 for maximum effectiveness in transformation (Reference OsborneOsborne, 2002). Then we apply, a natural log transformation to the shifted data. Table 1 shows that the transformed data presents reduced skewness across variables and the distributions are closer to normal. Although the transformed data still does not fully satisfy normality requirements based on the Shapiro-Wilk test (Reference Shapiro and WilkShapiro and Wilk, 1965), Fig. 1(b) indicates proximity to normal distributions. To represent the transformed data, we use tildes (~) instead of bars (−) over the variables.
Summary of the Descriptive Statistics
4.1.2. Clustering Analyses
In addition to the skewness, another important characteristic we observe in the data is a mix of human behaviors as opposed to uniform trend presented by the participants. Fig. 2(a) shows a scatter plot of participants’ rationality versus the proximity measure in Eq. 8. This plot visually reveals two distinct trends, making the analysis of the entire data as a whole uninformative, as opposing correlations cancel each other out. After testing several clustering methods, including k-means and density-based clustering, we use Gaussian mixture model to separate the two clusters as shown in Fig. 2(b), which provides the best visual separation. Gaussian mixture clustering is a probabilistic method that assumes data points are generated from a mixture of multiple Gaussian distributions, each representing a different cluster (Reference Fraley and RafteryFraley and Raftery, 1998). Gaussian Mixture Models assign probabilities to data points, allowing them to belong to multiple clusters simultaneously. In our case, we use hard clustering after assigning the probabilities, resulting in two distinct clusters. Both clusters contain a mix of winning and losing players, as well as decisions from both low-cost and high-cost treatments. There are no significant differences in the percentages of winning status or cost treatments between the two clusters. However, Cluster 1 comprises 30% of the overall dataset, while Cluster 0 accounts for 70%.
Comparison of Rationality vs Proximity to True Optimum in the Objective Space
4.2. Rationality and Proximity to the Optimal Solution
We examine the relationship between participants proximity to the true optimum and their rationality by analyzing the correlations between the
and
. We perform this analysis separately for the groups identified by the clustering approach. We present the scatter plots and the Pearson correlation metrics in Fig. 3. Although the results indicate similar behaviors in both the design and objective spaces, the objective space provides more robust results. In the experiment, F(x) changed in each round and the participants needed to base their strategies on the function response, i.e., the values in the objective space. As seen in Fig. 3, the points in the design space are spread over a wider area, resulting in slightly lower correlation coefficients. This behavior extends beyond rationality and proximity to other relationships. Thus, we present only the findings in the objective space in the remainder of the paper.
Correlation results between
and
Cluster 1 shows a direct correlation with r = 1.00 in the objective space, suggesting that participant decisions in that cluster approaches rational decisions as they find values closer to the true optimum in their search. On the other hand, Cluster 0 exhibits the opposite behavior, albeit with a weaker correlation (r = −0.33). The results remain statistically significant. Decisions in this cluster become more irrational as participants approach true optimum. The data used in this paper does not provide any further insight into the underlying cause of this contradictory behavior and we leave further analysis as a topic for future study with additional experiments. However, we show that human participants can employ a multitude of search strategies and an any future investigation must account for that fact.
4.3. Uncertainty and Its Impact on Rationality
To examine the impact of design space uncertainty as defined in Eq. 7 on rationality, we analyze the scatter plot between
$\mathop ˜\rho \limits^ _y $
and Ũ in Fig. 4. Similar to our analysis in Sec. 4.2, the behavior of clusters indicates a conflict: Cluster 1 shows a moderate positive correlation with r = 0.51, while Cluster 0 shows a weak negative correlation with r = −0.44. Both of these correlations are statistically significant. Although the correlation between
and Ũ is not as strong as that between
and
in Cluster 1, it still indicates that the decisions in that group tend to approach closer to rationality as uncertainty in the design space decreases. However, Cluster 0 directly contradicts this interpretation with an opposite behavior.
Correlation results between
and Ũ
4.4. Rationality and Competitive Success
Although BO with EI as a rational search process neglects the competition factor that exists in the experiment, we analyze the rationality of winning and losing participants to understand the trends in each group of participants. We show the box-plots of
for winners and losers in both clusters in Fig. 5. The figure shows that the median values and ranges of winners are significantly lower than those of losers in Cluster 1, suggesting that winners make decisions closer to rational ones in that cluster. However, there appears to be no clear distinction in Cluster 0 based on winning status.
Rationality comparison between winners and losers
To further analyze these results, we apply the Mann-Whitney U test to both clusters and across both cost treatments. The Mann-Whitney U test is a non-parametric statistical test used to determine whether there is a significant difference between the distributions of two independent groups (Reference Mann and WhitneyMann and Whitney, 1947). Unlike parametric tests, it does not assume normality and instead compares the ranks of the data, making it suitable for ordinal or non-normally distributed datasets. This approach is particularly relevant to this study since neither the raw nor the transformed data follow a normal distribution. Table 2 displays the p-values for the different tests conducted to compare the winning and losing participants in two clusters and cost treatments. In Cluster 1, results with p < 0.05 indicate that winners exhibit lower rationality measures than losers across both treatments, suggesting more rational choices. In contrast, in Cluster 0, only the low-cost treatment shows significant differences, while the high-cost treatment lacks consistency across clusters. These findings underscore the impact of cost treatment on rational decisionmaking. Given our rational process model does not explicitly account for the effect of competition in the search behaviors, it is expected the low-cost treatment to provide better differentiation since the participants may not feel the pressure of competition in this treatment as much as they do in high-cost treatment. When cost of search is higher, the participants may be more careful in their design decisions and a model that accounts for the competition may be necessary to understand this distinction.
Mann-Whitney U Test Results Comparing Winning and Losing Participants
5. Conclusion
In this paper, we investigated rational decision-making under uncertainty in engineering systems design based on the data from a 1 vs. 1 function optimization game. We used BO with EI as a baseline to assess participants’ behaviors. The findings of this study may have important implications for real-world design problems that exhibit the design characteristics mentioned in Sec. 2. Our results quantified how much each participantdeviated from purely rational search behaviors. Notably, the competition winners exhibited higher rationality. These results highlight an opportunity for designers to integrate Bayesian optimization-driven rational agents that can collaborate with human decision-makers, influencing their search strategies based on the design objective. For instance, these agents could guide designers toward unexplored regions of the design space, promoting solution diversity or navigate them toward exploitation, thereby reducing uncertainty through systematic knowledge acquisition (C1). Since uncertainty affects decision-making as shown in our study, such strategies could help designers make more informed and rational choices (C2). Additionally, our preliminary analysis suggests that cost influences participant rationality. When higher cost constraints are present, designers may be unable to fully explore the design space, potentially leading to suboptimal solutions (C3–C4).
This work did not include the impact of competition in the baseline rational model, focusing solely on developing metrics to assess individuals’ decision-making processes. Future studies should improve the baseline rational decision-making model with competitive factors, potentially using game theory. Also, the process in this paper is limited to individual decision-making whereas competitive design decisions in practice are made in teams. Extending the analysis to competition with design teams could further provide valuable insights that might be more applicable to design process followed in practice.
Acknowledgement
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-24-1-0265. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Air Force. Also, the authors gratefully acknowledge the financial support from the National Science Foundation through the grants CMMI-2321463 and CMMI-2419423.
Open access
