BWS and survey design

To measure the relative importance of potting mix characteristics among consumers, we implement a BWS approach, also known as maximum difference scaling. Developed by Finn and Louviere (Reference Finn and Louviere1992), BWS presents respondents with a list of three or more items and asks them to select the best (or most important) and the worst (or least important) of the items listed. The items can be product attributes, options of stores, or even opinions. Respondents’ attention span is limited, and they usually cannot effectively evaluate more than several items at a time. As a result, when there are too many items to include into a single list, the BWS exercise is usually repeated several times, each only includes a fraction of all the items to consider. Each of these lists is also referred to as a choice set. A statistical experimental design is often used to determine which items appear together in one choice set and how many times each item appears over all choice sets. For each choice set, the respondent provides their most and least preferred item, which identifies the maximum difference in their underlying value between the best and worst item, providing more accurate results of relative preferences than a standard Likert style question.

BWS is advantageous because it eliminates the bias of different interpretations of terms like “very important” or “somewhat agree” in rating scales (Aizaki and Fogarty, Reference Aizaki and Fogarty2023). It also forces tradeoffs in that a person cannot select “very important” for all items. As well, BWS has better predictive validity compared to several other rating techniques (Chrzan and Golovashkina, Reference Chrzan and Golovashkina2006; Mielby et al., Reference Mielby, Edelenbos and Thybo2012). We implement the BWS object case (known as Case 1), which measures preference intensity for a list of objects, meaning a list of attributes, characteristics, traits, or issues to be measured. Hereafter we refer to the items in a list as characteristics.

Two other cases of BWS exist. Case 2 is known as profile BWS in which multiple levels exist for each attribute (e.g. the size of the bag of potting mix (attribute) can be small, medium, or large (levels)), with the respondents selecting their most and least preferred of the attribute level of those shown. While more flexible, its validity is questionable if attributes shown increase or decrease consumer values (Soekhai et al., Reference Soekhai, Donkers, Levitan and de Bekker-Grob2021). Case 3 is the multi-profile BWS in which the respondent selects among several profiles of items, each containing its own combination of attributes and levels, also known as Discrete Choice Experiments, are commonly used in consumer preferences for food and horticultural products (Chavez et al., Reference Chavez, Palma, Byrne, Hall and Ribera2020; Hu et al., Reference Hu, Woods and Bastin2009). While DCEs allow for the calculation of economic value, such as Willingness to Pay, they face several challenges. First, their complexity makes them less intuitive to understand and answer compared to Case 1 and 2 BWS. Further, when calculating Willingness to Pay, DCEs are known to suffer from hypothetical bias, despite methods to ameliorate this issue (Penn and Hu, Reference Penn and Hu2018, Reference Penn and Hu2023). Further, their increased complexity reduces the likelihood of respondent comprehension (Pearce et al., Reference Pearce, Harrison, Watson, Street, Howard, Bansback and Bryan2021) and/or requires additional instructions to facilitate comprehension, thus increasing survey length and cost of data collection.

We consider eleven potting mix characteristics. Table 1 lists these characteristics and the descriptions of each shown to respondents, which are: Contains fertilizer, Packaging, Moisture control, Organic, Contains peat moss, Plant/Garden-specific formulation, Home delivery, Locally owned retailer, Brand, Price, and Pre-mixed. These characteristics were selected through several rounds of discussions with potting mix producers, retailers, and consumers. Some of these characteristics were chosen because they are common labels on potting mix bags or have been examined, though in a more piecemeal fashion, in previous research (Campbell et al., Reference Campbell, Rihn and Khachatryan2020; Dahlin et al., Reference Dahlin, Beuthner, Halbherr, Kurz, Nelles and Herbes2019; Herbes et al., Reference Herbes, Dahlin and Kurz2020; Thomas et al., Reference Thomas, Jensen, Lambert, English, Clark and Walker2021; Thomas, Reference Thomas2019). Home delivery and whether the potting mix is purchased from a Locally owned retailer are two characteristics which have not been previously examined. The former describes a recently available method for consumers to acquire potting mix (whether as loose, bulk material or bagged) and is included as a characteristic to see how convenience of delivery is preferred to other characteristics. The localness of a retail business has been viewed favorably in other product segments such as food (Printezis et al., Reference Printezis, Grebitus and Hirsch2019; Soley et al., Reference Soley, Hu and Vassalos2019), and it may have a similar appeal among gardeners who prefer to purchase from locally owned stores.

Table 1.

List of BWS objects and its description shown in the survey

Several characteristics were considered but ultimately combined into another characteristic to reduce the number of characteristics in the BWS (and therefore the number of choice sets) answered by respondents. Compost is a type of organically sourced fertilizer or plant food, characteristics already included in this study, so compost is excluded, and “no artificial or synthetic components” is in the definition of Organic, therefore it is eliminated as well. Formulated for specific plants and types of gardens was consolidated into Plant/Garden-specific formulation, which encompasses both characteristics.Footnote ²

The BWS was designed in R using support. BWS (Aizaki and Fogarty, Reference Aizaki and Fogarty2023). We employed a Balanced Incomplete Block Design (BIBD), which requires each object to appear the same number of times, and the probability of joint appearance of two objects is equal across all objects, meaning that there is balance and orthogonality between each object (Flynn and Marley, Reference Flynn, Marley, Flynn and Marley2014). By construct, the BIBD has a 100% D-efficiency. We verified that our BWS design of eleven characteristics is a BIBD, with each person answering eleven choice sets of five characteristics per choice set. The order of choice sets and order of characteristics within each choice set was randomized.

Each respondent was asked to select the most important and least important potting mix characteristic of the five options within each choice set. At the start of the BWS section, following Caputo and Lusk (Reference Caputo and Lusk2020), respondents received the following instructions: “The following 11 questions are about your preferences regarding potting mix characteristics and availability. Each question is composed of five characteristics that apply to potting mixes available in the US, and we would like to know which characteristic you prefer most and which characteristic you prefer least of the options available in each question.” The respondents also saw a short description of each characteristic. An example choice set appears in Figure 1.

Figure 1.

Example BWS choice set.

The BWS was embedded as part of a larger survey of gardeners within the US, which contained six sections. The design of the survey is based on questions and elements seen in related works (Campbell et al., Reference Campbell, Rihn and Khachatryan2020; Dahlin et al., Reference Dahlin, Beuthner, Halbherr, Kurz, Nelles and Herbes2019). The first section contained two screening questions: “do you currently have any plants or garden space you take care of?” and “have you purchased potting mix or soil within the past year?” Only those who answered “Yes” to both questions took the full survey, ensuring that the respondent was part of the target population of gardeners/users of potting mix with some recent knowledge of gardening and potting mix.

Section 2 focused on basic characteristics of the respondent’s participation and experience in gardening in terms of the size of their garden, types of plants, and participation in gardening related organizations/clubs. Questions include: “how long have you participated in gardening of any type,” “which resources have you used in the past year to learn about gardening,” “what size is your garden,” “which plant hardiness/grow zone is your garden space located,” “what is the 5-digit zip code of your gardening space,” and “are you in any gardening related organizations?”.

Section 3 focused on respondents’ purchases and preferences for potting mix. Questions include the number of bags of potting mix purchased and type of retailer purchased from, preferred brand, bag size, formulation, or desirable potting mix features. These questions allow for an alternative gauge of consumer behavior to compare BWS results against and primes respondents to think about their preferences prior to answering the BWS. Section 4 contained the BWS questions.

Section 5 focused on peat moss and its connection to the environment. Respondents were asked about their familiarity with peat moss and if they had knowingly used potting mixes containing peat moss in the past. They then provided Likert-style responses on the environmental effects of their gardening products and plants, their willingness to switch to alternative products/practices, and their perceived harmfulness of using potting mix ingredients: perlite, peat moss, compost, and vermiculite. The last part of this section directed respondents to a small information treatment about potential issues of using peat moss, the focus of a separate study.

The final, sixth section asked for demographic information and other characteristics of the respondents such as whether they are master gardeners or master naturalists. These characteristics are useful in terms distinguishing how preferences for potting mix may vary across groups of gardeners. For example, master gardeners and master naturalists have more experience and skills in gardening and environmental/natural resource conservation than an average consumer. These two types of respondents may value potting mix characteristics differently from the average public. Definitions of gardening, attitudinal, and demographic characteristics used in the analysis appear in Table 2.

Table 2.

Variable definitions and descriptive statistics (n = 499)*

*Based on those who answered the question and did not select “Prefer not to answer”

The survey was conducted online using the Qualtrics platform. Recruitment of participants occurred through convenience sampling via recruitment though social media (Facebook, Instagram, Twitter, and LinkedIn) of eleven related organizations and businesses,Footnote ³ with several sending out/posting reminders. Each organization’s primary audience and members (though not exclusively) are in Louisiana. The formal survey was distributed from August to September 2023 and closed with 671 responses. The survey uses several data quality checks to remove bot responses, duplicate responses from the same individual, and inattentive responses,Footnote ⁴ all issues shown to affect survey data quality results (Gao et al., Reference Gao, House and Xie2016; Goodrich et al., Reference Goodrich, Fenton, Penn, Bovay and Mountain2023).

Statistical analysis

To analyze the BWS results, we use the aggregated counting method from Aizaki et al. (Reference Aizaki and Fogarty2023), which provides a score for each item based on the number of times it was selected the best and worst across all respondents. Each potting mix characteristic (i) is tallied in terms of the number of times it is selected best or worst and the cumulative points allow the characteristics to be ranked. The frequency in which a characteristic is chosen as the best summing across all respondents is shown as B _i , and the frequency in which a characteristic is chosen as the worst summing across all respondents is shown as W _i . Subtracting W from B yields the Best-Worst score (BW _i ),Footnote ⁵ the aggregated score for each characteristic from all respondents, as shown in Equation (1).

(1) $$BW_{i}=B_{i}-W_{i}$$

The more positive (negative) BW _i , the more (less) preferred characteristic i is for respondents. Equation (2) shows the standardized score (std.BW _i ), which divides BW _i by N, the total number of responses, times r, the number of times that characteristic occurs in the survey. Quantity std.BW _i can range from −1 to 1 across all characteristics, facilitating comparison.

(2) $$std.BW_{i}=BW_{i}/(N*r)$$

Let variable X be the indicator for a characteristic. It only equals one if the characteristic appears in a choice set. Let β be the associated coefficients of X. We define Equation (3) as an indicator function for respondent n choosing the j-th characteristic as the best or the most preferred characteristic and the k-th characteristic as the worst or least preferred characteristic plus a random, unobservable component (e), made among all characteristics J shown in the t-th choice set:

(3) $$I_{njt}={X_{njt}}\beta _{j}-{X_{nkt}}\beta _{k}+e_{njt}$$

For a potting mix characteristic not included in a choice set (X = 0), the characteristic and its associated coefficient will not appear in the indicator functions associated with that choice set; furthermore, indicator I is zero if either j or k are not present. As such, in the following discussion, dropping variable X will not affect the expressions. Equation (3) is maximized when the difference between the two chosen characteristics is the largest among all possible differences. Alternatively, the probability of observing these choices is the probability that the difference between characteristic j and k is greater than all other J(J-1)-1 possible differences in choice set t (Lusk and Briggeman, Reference Lusk and Briggeman2009).

Assuming that e _nj is distributed iid type I largest extreme value, then estimates β of the best and worst characteristic (j and k), relative to an omitted reference characteristic normalized to zero, can be estimated according to the conditional logit model. To allow for random taste heterogeneity across individuals n, we instead rely on a mixed logit model, as in Equation (4).

(4) $$P_{n}(best=j,\textit{worst}=k)=\int _{\beta }{\exp \left(\beta _{nj}-\beta _{nk}\right) \over \sum _{l,m=1\;and\;l\neq m}^{J}\exp \left(\beta _{nl}-\beta _{nm}\right)}f(\beta _{n})d\beta _{n}$$

We conduct a simulated maximum likelihood estimation based on 1000 Halton draws, assuming that preference variation follows a normal distribution for all characteristics. Relying on this approach also allows for the calculation of the share of preferences, which measures the relative importance of characteristic i (Lusk and Briggeman, Reference Lusk and Briggeman2009), as in Equation (5).

(5) $$SP_{i}={\exp (\hat{\beta }_{i}) \over \sum _{j=1}^{11}\exp (\hat{\beta }_{j})}$$

The share of preferences across all eleven attributes must sum to one. Comparing the ratio of SP across attributes portrays relative importance. For example, if SP _i = 0.2 and SP _j = 0.1, then attribute i is twice as important as attribute j (0.2/0.1 = 2.0). All estimation was completed in R 4.3.1 and using the “Support.BWS” package (Aizaki and Fogarty, Reference Aizaki and Fogarty2023).