2.1. Design methods and challenges for business experimentation in CBM

Business experimentation in the context of CE is defined as an iterative approach to developing and verifying circular value propositions in a real market environment with customers and stakeholders, providing evidence of their feasibility through learning based on empirical information acquisition (Reference Bocken, Weissbrod and AntikainenBocken et al., 2021). As specific design methods for business experimentation, methodologies such as Effectuation and Lean Startup are widely referenced in the context of CBMs (Reference Konietzko, Baldassarre, Brown, Bocken and HultinkKonietzko et al., 2020). Effectuation is an approach that maximizes the utilization of limited available resources and information. Under this approach, the subjects of experiments are determined based on available means, such as “who I am,” “what I know,” and “whom I know,” as well as criteria such as “affordable loss” (Reference Konietzko, Baldassarre, Brown, Bocken and HultinkKonietzko et al., 2020; Reference Bocken and CoffayBocken & Coffay, 2023). Lean Startup is an approach for verifying hypotheses through rapid “Build-Measure-Learn” iterative cycles, where the subjects of experiments are determined based on criteria such as positioning within the competitive environment and expected returns (Reference Konietzko, Baldassarre, Brown, Bocken and HultinkKonietzko et al., 2020; Reference Bocken and CoffayBocken & Coffay, 2023).

However, it has been pointed out that in these existing methods, the selection of experimental conditions is qualitative, and participants often tend to make judgments and decisions following intuition rather than the decision criteria defined at the experimental design stage (Reference Konietzko, Baldassarre, Brown, Bocken and HultinkKonietzko et al., 2020; Reference Bocken, Weissbrod and AntikainenBocken et al., 2021). Since CBMs require verification over a long-term timeline involving product collection and recirculation compared to Linear Business Models (LBMs), business experimentation in CBMs necessitates greater costs and time (Reference Linder and WillianderLinder & Williander, 2017). Therefore, in business experimentation for CBMs, a quantitative design method is required to optimize the experimental design regarding which experimental conditions should be verified and in what order, thereby acquiring information to efficiently reduce uncertainty with a limited number of experiments.

2.2. Bayesian optimal experimental design

BOED frames experimental design as a decision-making problem aimed at maximizing the expected utility of an experiment (Reference Chaloner and VerdinelliChaloner & Verdinelli, 1995). Drawing on Reference LindleyLindley’s (1956) distinction between decision-making and information acquisition, BOED prioritizes the latter. In this framework, knowledge regarding unknown parameters is represented as a probability distribution. A prior distribution based on existing knowledge is established before the experiment, and upon obtaining observational data from the experiment, it is updated to a posterior distribution using Bayes’ theorem (Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024). The value of an experiment in BOED is evaluated by the expected utility calculated based on this posterior distribution, and the experimental condition maximizing this value is derived as the optimal experimental design (Reference Chaloner and VerdinelliChaloner & Verdinelli, 1995).

A widely used utility metric is EIG, which uses Shannon information theory to quantify the reduction in parameter uncertainty (Reference Chaloner and VerdinelliChaloner & Verdinelli, 1995; Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024). To estimate EIG, an observation model is required to describe how observational data is generated for arbitrary experimental conditions and unknown parameters. In problem settings where handling the likelihood analytically is difficult due to the nonlinearity or high dimensionality of the target phenomenon, an approach involving the construction of a simulation model to reproduce the phenomenon and positioning it as the observation model is widely employed (Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024). However, in simulation-based BOED, since the value of EIG cannot be calculated analytically, approximate estimation using methods such as the Monte Carlo method is generally required. Consequently, in the process of searching for optimal experimental conditions, the computationally expensive estimation of EIG must be repeated for numerous candidates, resulting in an enormous computational load (Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024; Reference Coons and HuanCoons & Huan, 2025). To address this issue of computational cost, the introduction of a surrogate model that approximates the input-output relationship of the simulation model is a common method in simulation-based BOED (Reference Coons and HuanCoons & Huan, 2025). Among such models, Gaussian Process Regression (GPR) is generally adopted as a surrogate model for BOED, given its ability to flexibly approximate nonlinear input-output relationships from a small amount of training data, while simultaneously estimating not only the mean of predicted values but also the uncertainty of the prediction (Reference Greenhill, Rana, Gupta, Vellanki and VenkateshGreenhill et al., 2020; Reference Pandita, Tsilifis, Awalgaonkar, Bilionis and PanchalPandita et al., 2021).

BOED can be extended to adaptive experimental design, which sequentially searches for the next optimal experimental condition while incorporating past experimental results in a series of experimental processes, rather than being limited to single-shot experiments (Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024). In recent years, accompanying improvements in computing power, the application of BOED has advanced in diverse fields such as psychology, physics, and materials discovery, and it is also being applied to market research and user modeling in the business domain (Reference Greenhill, Rana, Gupta, Vellanki and VenkateshGreenhill et al., 2020; Reference Rainforth, Foster, Ivanova and SmithRainforth et al., 2024). However, research regarding the application of BOED in the design of business experimentation for CBMs remains limited.

2.3. Positioning of the study

As noted in the preceding sections, the transition to CBMs entails greater uncertainty regarding customer product usage duration, product condition upon return, and market acceptance compared to conventional LBMs. While iterative learning through business experimentation is indispensable for reducing these uncertainties, the inherent characteristics of CBMs present a challenge, requiring extended periods for verification and potentially increasing experimental costs. However, existing design methods for business experimentation, such as Effectuation and Lean Startup, rely on practitioner intuition and qualitative judgment for the selection of experimental conditions. Consequently, limitations exist regarding the optimization of experimental efficiency in terms of determining which experimental conditions should be verified and in what order within limited resources. Although BOED, which derives experimental conditions that most efficiently reduce uncertainty through the maximization of EIG, is considered useful for addressing this issue, research applying BOED in the context of CBM business experimentation design remains limited. Therefore, this study proposes a design method for business experimentation using BOED as a new approach to reduce uncertainties regarding CBMs. This method enables the selection of experimental conditions that reduce uncertainty most efficiently with limited resources, thereby supporting the implementation of CBMs.

3.1. Flow of the proposed methodology

In Phase 1, a simulation model reproducing the target market in CBM business experimentation is constructed, and a comprehensive input-output dataset for surrogate model construction in Phase 2 is created. As input variables, controllable design variables $$\bf{\it{d}}$$ , such as product price, which the provider aims to verify and determine through business experimentation, and uncontrollable unknown parameters $$\bf{\theta }$$ , such as failure rates, which act as sources of business uncertainty, are employed. As output variables, objective variables $$\bf{\it{y}}$$ , such as average annual profit, which evaluate business success or failure, are used. Here, it is assumed that the values and distributions taken by the design variables and unknown parameters are discrete. In this phase, simulations are executed for all assumed combinations of diverse design variables and unknown parameters, and objective variables corresponding to each input pair are calculated.

In Phase 2, based on the input-output dataset created in Phase 1, a surrogate model that rapidly approximates the input-output relationship of the simulation model is constructed. Here, GPR is adopted as the surrogate model for simulation-based BOED. In this study, independent GPR models are constructed for $$R$$ objective variables $${y_1}, \ldots ,{y_R}$$ . The GPR for each objective variable $${y_r}$$ ( $$r = 1,\;\; \ldots ,\;R$$ ) assumes that the output for input ( $$\bf{\it{d}},\;\;\bf{\theta }$$ ) follows a Gaussian process defined by a mean function $${{\rm{\mu }}_r}\left( \bf{\it{d}},\;\;\bf{\theta } \right)$$ and a variance function $${\rm{\sigma }}_r^2\left( \bf{\it{d}},\;\;\bf{\theta }\right)$$ . Prior to model training, input features, namely the design variables and the unknown parameters, and the objective variables are normalized to follow a standard normal distribution. The generalization performance of the constructed surrogate model is evaluated by cross-validation using the $${R^2}$$ score as the evaluation metric.

In Phase 3, using the surrogate model constructed in Phase 2, the optimal experimental condition $$\bf{\it{d^{*}}}$$ is selected based on the BOED framework. Specifically, for each experimental condition $$\bf{\it{d}}$$ in the experimental condition space , the EIG is approximately estimated by the Monte Carlo method, and the condition maximizing this is identified. In this study, it is assumed that the unknown parameter takes $$M$$ possible values , and its prior probability is defined as a probability mass vector $$\bf{\pi} $$ where the sum of probabilities assigned to each candidate is 1. The specific selection process for the optimal experimental condition $$\bf{\it{d^{*}}}$$ is shown below (Figure 1).

Figure 1.

Algorithm for selecting an optimal experimental condition

First, the unknown parameter $${{\bf{\theta}} ^{\left( m \right)}}$$ is sampled based on the prior probability mass vector. Next, the unknown parameter $${{\bf{\theta}} ^{\left( m \right)}}$$ and the experimental condition $$\bf{\it{d}}$$ are input into the surrogate model, and a virtual observation $${y_r}$$ is generated. A Bayesian update is performed using this observed value $${y_r}$$ , and the reduction in uncertainty of the unknown parameter is evaluated as the change in entropy. This process is repeated $$N$$ times, and by averaging the results, the expected value of EIG for the experimental condition $$\bf{\it{d}}$$ is estimated. Finally, to aggregate multiple objective variables, a global index $$EI{G_{combined}}\left( \bf{\it{d}} \right)$$ is defined by averaging the EIG of each objective variable with equal weights, and the experimental condition $$\bf{\it{d^{*}}}$$ maximizing this is selected.

3.2. Detailed explanation of the simulation model

3.2.1. Simulation structure

The simulation model comprises the following four modules (Figure 2).

The Ecosystem Module is responsible for the control and coordination of the entire simulation. This class initializes the customers, manufacturers, PaaS providers, products, and business models participating in the ecosystem, and manages each module during the monthly simulation cycle. Details regarding the monthly simulation cycle are described in the subsequent “Simulation flow” section. The Stakeholder Module models the behaviors of the key economic agents: manufacturers, PaaS providers, and customers. Manufacturers conduct product production and price calculations based on demand. PaaS providers procure products and calculate subscription prices according to usage periods and plans. Customers possess individual planned usage periods and select products and plans. The Product Module manages the lifecycle and status of individual products. Specifically, the module tracks the maximum lifespan, elapsed months, usage count, and allocation status of products, while executing monthly status updates, availability determinations, and failure assessment. The failure model implements two types of failures: initial failures upon introduction and age-related failures based on the Weibull distribution. Products are disposed of upon exceeding the maximum lifespan or maximum usage count, with relevant history and material flows recorded to ensure traceability. The Business Model Module calculates financial flows among stakeholders. Specifically, revenue, costs, and profits are calculated based on revenue-sharing configurations between the PaaS provider and the manufacturer. The manufacturer’s revenue consists of procurement revenue from the PaaS provider and the distributed portion of subscription revenue, while the PaaS provider retains the post-distribution subscription revenue. Costs include production, procurement, and labor expenses.

Figure 2.

Class diagram

3.2.2. Simulation flow

The simulation is executed in three stages: the initialization phase, the simulation phase, and the evaluation phase. First, in the initialization phase, the Ecosystem Module defines and instantiates the roles and attributes of each entity. The simulation phase is executed in a monthly cycle (Figure 3).

Figure 3.

Activity diagram

Figure 3 Long description

The flowchart illustrates the stages of a product-as-a-service (PaaS) business model, detailing the interactions between the PaaS provider, customer, and manufacturer. The process begins with monthly initialization, where the PaaS provider starts the monthly cycle, checks available products, and creates new customers. This is followed by product matching, where the PaaS provider executes optimal matching and checks for initial malfunctions. If a malfunction occurs, the product is disposed of, and a shortfall is ordered. The manufacturer then produces the products, which are procured and allocated to customers. The product lifecycle involves customers using the products, notifying the provider of malfunctions, and returning products at the end of the usage period. The PaaS provider repairs and returns products to inventory. The financial calculation stage involves calculating the revenue and cost for both the PaaS provider and the manufacturer. The flowchart includes decision points for matching success, initial malfunctions, product malfunctions, and the end of the usage period, with corresponding actions and outcomes.

First, in Monthly Initialization, the PaaS provider assesses the available product inventory, and simultaneously, new customers are generated based on demand data. In Product Matching, the PaaS provider executes optimal matching based on new customers and the available inventory list. Here, customers select the product that is most desirable and possesses a remaining useful life closest to their own planned usage period from among the available products. Product Allocation is conducted in two stages. First, products with a remaining useful life exceeding the customer’s planned usage period are searched for within the existing inventory. If a suitable product is found, it is assigned to the customer following a probabilistic initial failure determination. Products identified as initially defective are disposed of, and the corresponding customers are carried over to the next month. If allocation is not possible from inventory or if the product is defective, the manufacturer produces new products, and the PaaS provider procures products for a second allocation attempt. In this instance, initial failure determination is performed similarly; defective products are disposed of, and the corresponding customers are carried over. In the Product Lifecycle, the usage duration of all products currently in use is first updated. Products whose usage period has concluded are returned to the PaaS provider and sorted into inventory or disposal according to their remaining useful life. Additionally, the elapsed months and usage months of each product are updated, and age-related failure is probabilistically determined based on the Weibull distribution. Repair is attempted for failed products; if a product is deemed irreparable, it is disposed of after recording the disposal reason and status.

Finally, in Financial Calculation, the PaaS provider and the manufacturer calculate their respective revenues and costs for the current month. If the simulation period has not concluded, the next monthly cycle is initiated.

Upon completion of the entire simulation period, the process transitions to the evaluation phase. Results obtained from each module are aggregated to calculate indicators such as financial metrics for each stakeholder, product disposal volume, failure rate, matching success rate, and initial failure rate.

4.1. Case study overview

This case study focuses on a CBM in which a manufacturer and a PaaS provider collaborate to provide an air conditioner subscription service. As this service has not yet been widely adopted, it is characterized by significant uncertainty regarding unknown parameters such as customer preferences. Consequently, there is a critical need for business experimentation, making it a suitable case for applying the proposed method. The CBM examined in this case study employs a revenue-sharing model in which revenue is distributed between the manufacturer and the PaaS provider; a fixed proportion of the monthly usage fees collected from customers is allocated to the manufacturer, while the remainder is retained by the PaaS provider.

The manufacturer bears product manufacturing costs and labor expenses. Conversely, the PaaS provider bears product procurement costs, assumed to be equal to the manufacturer’s production costs, as well as repair and transportation costs. This case study involves five air conditioner models. The product grades, price ranges, and assumed selection rates for these five models are presented below (Table 1).

Table 1.

Specifications and selection ratios of air conditioner models

Each product possesses common specifications consisting of a product lifespan of 120 months and a maximum usage count of four times. Furthermore, an initial failure rate and an age-related failure model based on the Weibull distribution are configured. Regarding the service structure, the PaaS provider offers two distinct subscription plans. Plan $${\rm{\alpha }}$$ is characterized by the absence of a minimum contract period and the requirement of a delivery fee. Additionally, this plan offers long-term incentives, specifically a 70% discount for usage durations of three months or longer and a 50% discount for durations of 12 months or longer. In contrast, Plan $${\rm{\beta }}$$ stipulates a minimum contract period of 12 months and exempts the delivery fee, while providing a 50% discount for usage exceeding 12 months.

The number of customers was established as a monotonically increasing fixed value accounting for business growth, based on interviews with business practitioners. In this case study, the primary unknown parameters that practitioners seek to clarify through business experimentation comprise three elements: the average customer usage duration, the plan selection rate, and the initial failure rate. Furthermore, the design variables that practitioners aim to verify through business experimentation are twofold: the price multiplier and the lineup of offered models. The objective of this case study is to support the implementation of CBMs by efficiently reducing uncertainty regarding unknown parameters within a limited number of experiments.

4.2. Flow of the case study implementation

In Phase 1, the simulation model detailed in Chapter 3 was implemented by tailoring it to the air conditioner PaaS business scenario established in Section 4.1. The simulation duration was set to 120 months. In this case study, the potential values and distributions of the design variables and unknown parameters, as well as the objective variables, were configured as follows based on interviews with business practitioners.

First, 124 experimental conditions $$\bf{\it{d}}$$ are defined by two design variables: a price multiplier of 0.8, 1.0, 1.2, or 1.4, and an offered model lineup consisting of 31 combinations of five available options. Next, 12 unknown parameter scenarios $$\bf{\theta} $$ are derived from three factors: an average usage duration of 12 or 24 months, a plan selection rate (Plan $${\rm{\alpha }}$$ : Plan $${\rm{\beta }}$$ ) of 3:7, 5:5, or 7:3, and an initial failure rate of 1% or 2%. A uniform prior probability $$\bf{\pi} $$ of 1/12 is assumed for each scenario. Finally, four objective variables $$\bf{\it{y}}$$ evaluate performance: average annual profit over the final five years and time-to-profitability for both the PaaS provider and the manufacturer. In this case study, using the constructed simulation model, a total of 1,488 simulations, representing the combination of all 124 experimental conditions and all 12 parameter scenarios, were executed to generate an input-output dataset.

In Phase 2, independent GPR models were constructed for each of the four objective variables using the 1,488 input-output data points generated in Phase 1. An RBF kernel with Automatic Relevance Determination (ARD), capable of automatically learning the importance of each input variable, was employed, and hyperparameters were optimized to maximize the log marginal likelihood. To evaluate the generalization performance of the constructed surrogate models, cross-validation using the $${{\rm{R}}^2}$$ score was conducted. Since values of $${{\rm{R}}^2}$$ > 0.95 were confirmed for all objective variables, the surrogate models were considered sufficiently accurate for EIG estimation.

In Phase 3, experimental conditions maximizing the EIG for the unknown parameters were selected based on the BOED framework described in Chapter 3. The number of Monte Carlo samples was set to $$N\; = 500$$ , and $$EI{G_{combined}}\left( \bf{\it{d}} \right)$$ , calculated as the equal-weighted average of the EIGs for the four objective variables $${y_r}$$ , was employed as the evaluation metric.

$$EI{G_{combined}}\left( \bf{\it{d}} \right)$$ was estimated for all 124 experimental conditions ( ), and the top five experimental conditions yielding the highest values are presented below (Table 2).

Table 2.

Top 5 design variables with the highest combined EIG

Distinct characteristics were observed in the top five experimental conditions selected via BOED. The price multipliers were polarized toward either the lowest value of 0.8 or the highest value of 1.4, whereas intermediate values of 1.0 and 1.2 were not included in the top five rankings. Regarding the offered products, the top rankings were dominated by either configurations centered on low-price models, specifically Product A, or conversely, configurations restricted to high-price models such as Products C, D, and E. These results suggest that in future business experimentation, uncertainty can be efficiently reduced not by comprehensively verifying all experimental conditions, but by limiting the price multipliers to 0.8 and 1.4, and by targeting both low-price-centered and high-price-centered product configurations for examination.