2.1. Theoretical issues

In this subsection, the identified issues related to the theoretical development of the UX concept, its definitions, its elements and interpretations are synthesised and described.

The difficulty in defining UX boundaries concerning the concept of ‘usability’ is reported by several scholars (Følstad & Rolfsen Reference Følstad and Rolfsen2006; Sauer, Sonderegger, & Schmutz Reference Sauer, Sonderegger and Schmutz2020) as soon as the term ‘UX’ was introduced by Norman, Miller, & Henderson (Reference Norman, Miller and Henderson1995). Donald Norman claimed that UX goes far beyond the limited concept of ‘Usability’, justifying the introduction of such a new term. More than two decades later, however, the versatility of the meaning of ‘UX’ led to an overuse of this concept, which lost its initial strength and significance. Actually, Norman himself complained about the abuse of his term (Merholz Reference Merholz2007), which is sometimes misused or is not the real focus of the research if ‘UX’ is considered formally. The lack of clarity around the term ‘UX’ was one of the first issues that emerged from the literature, which caused fragmentation in the UX research (hereinafter, issue terminology).

To better understand UX, Berni and Borgianni Berni & Borgianni (Reference Berni and Borgianni2021b) summarised the various key elements of UX, making order among the fragmentary definitions in the literature. The scholars addressed this fragmentation in terms of conceptual contributions proposing interpretations, definitions and theoretical frameworks of UX. Key components (user, interaction, system and context) and dimensions (ergonomic, affective and cognitive experiences: definitions thereof are given in Section 3) were identified. Components and dimensions were categorised into fundamental elements and influence factors, with a particular focus on their impact on the interaction output. The authors strongly suggest analysing both key components and UX dimensions in a holistic way, since most of the studies consider those elements separately instead of as different variables affecting the same experience. Otherwise, the risk is to focus only on certain key components while neglecting others. This is the case of the context where the interaction takes place. While elements such as ‘user’ and ‘system’ are often considered, ‘context’ is largely neglected despite its acknowledged importance in UX. The reason could lay behind the different ways in which context can be interpreted, namely at least the following four dimensions according to Berni & Borgianni (Reference Berni and Borgianni2021b).

1. 1. Cultural. Cultural studies interpret and analyse the cultural background of users to understand how differences in language, habits and norms affect the interaction with the product (Ortiz & Aurisicchio Reference Ortiz and Aurisicchio2011).

2. 2. Social. Forlizzi (Reference Forlizzi2008) stresses the importance of the social aspect of the context as interaction of multiple actors with a product. Focusing on this interpretation of the context could help designers understand and think about the dynamics of social behaviour while designing product (Ortiz & Aurisicchio Reference Ortiz and Aurisicchio2011). The HPI includes various elements, which are overall ascribable to the psychological and social reality, the so-called Umwelt (Carbon Reference Carbon2020), which therefore includes cultural aspects too.

3. 3. Physical/spatial. The space factor can be intended as both physical surrounding (featured by the location, light conditions, temperature and so on) and the technology used to display the interaction itself, for example, virtual or physical interaction (Rebelo et al. Reference Rebelo, Noriega, Duarte and Soares2012).

4. 4. Temporal. Interactions are strongly affected by the timing of the HPI, including the experience accumulated over time, leading to different expectations (Becattini et al. Reference Becattini, Borgianni, Cascini and Rotini2017). The consideration of the temporal context in HPI is one of the main challenges of UX in design, as explained below.

One way to study the context according to a temporal interpretation is to consider UX as a dynamic concept that changes over time (Kujala et al. Reference Kujala, Roto, Väänänen-Vainio-Mattila, Karapanos and Sinnelä2011) – an approach which turned out fruitful also in the related area of art experience (Muth & Carbon Reference Muth and Carbon2016; Muth, Raab, & Carbon Reference Muth, Raab and Carbon2017; Muth, Hesslinger, & Carbon Reference Muth, Hesslinger and Carbon2018). This can be done in several ways:

• • Studying different kinds of interactions at different phases of use. Yoon, Kim, & Kang (Reference Yoon, Kim and Kang2020) divided the interaction into seven phases: before purchase, during purchase, unboxing, first use, familiarisation, use, disposal or repurchase. The scholars have highlighted a substantial diversity in the positive experiences of users according to the phase of product use. To study UX, scholars employed retrospective techniques. Participants were asked to recall and evaluate an experience rather than interact with the product directly. This led participants to filter the evaluation through their memory of a past event that may be affected by emotional biases and inaccuracies. In other studies, the experience is not recalled from the past but is projected in the future through scenario building. Participants must imagine a situation and project their potential interaction using future-scenario techniques (Sleeswijk Visser & Visser Reference Sleeswijk Visser and Visser2006) (hereinafter, issue retrospective/projected).

• • Considering the context as the user’s background, with their preferences and expectations. UX is based on expectations, associations and knowledge, as all kinds of experiences are very subjectively driven. Expectations, which are largely overlooked in UX research, play a role in the mental representation of the product experience; these are plainly intertwined with people’s knowledge and experience, stimulated by what is sensed out of the product, and consequently transformative (Pettersson Reference Pettersson2017).

• • Studying the context as Zeitgeist factor (Carbon Reference Carbon2011). UX of products is modulated by Zeitgeist-dependent effects which play a major role when UX is investigated over a more extended period (see Carbon Reference Carbon2011) – and such longer perspectives are important to understand the usage and experience of products.

While efforts to address UX in a holistic way are scarce, experiences are more diffusedly classified as positive or negative. Some scholars focus on the positive–negative distinction to understand which output should be used to improve a product within the design process. Therefore, positive and/or negative experiences have been extensively studied in the literature (Hassenzahl Reference Hassenzahl2010; Kim Reference Kim2012; Kim & Christiaans Reference Kim and Christiaans2012; Fokkinga & Desmet Reference Fokkinga and Desmet2013; Yoon, Pohlmeyer, & Desmet Reference Yoon, Pohlmeyer and Desmet2017; Fokkinga, Desmet, & Hekkert Reference Fokkinga, Desmet and Hekkert2020; Yoon, Kim, & Kang Reference Yoon, Kim and Kang2020), even though scholars do not always agree on which of the two aspects of UX is the most useful in design. For example, Kim & Christiaans (Reference Kim and Christiaans2012) and Kim (Reference Kim2014) developed a framework focusing on an empirical cross-cultural study aimed at understanding which users’ characteristics related to products can lead to users’ negative experiences. This work is helpful in foreseeing unwanted negative experiences to avoid them. According to Yoon, Kim, & Kang (Reference Yoon, Kim and Kang2020), it is better to seek the elements that arouse a positive experience, rather than just mitigating or avoiding unwanted ones. This view is strengthened by Hassenzahl (Reference Hassenzahl2010), who claims that eliminating suffering and frustrations caused by undesired features can improve a product but it does not ensure a positive experience.

Three dimensions of experience can be identified beside the general division between ‘positive’ and ‘negative’. Those dimensions have been identified as ergonomic, cognitive and emotional experience (details are provided in Section 3) depending on whether the focus is on the system, its features and functionality, on the user and how it perceives the system or which feelings arise from the interaction between the user and the system.

Each of these types of experiences is characterised by the focus on specific aspects that come from the HPI resulting in different approaches used to investigate UX in the literature. The main approaches are three and they somehow correspond to the main experiences described above.

• • The ergonomic approach is strongly related to the functional side of the HPI and lined to the concept of affordances (Crilly Reference Crilly2011; Carbon Reference Carbon2019b).

• • The user-centred design (UCD) approach is closely connected to the users, their cognitive perception of a product and their involvement in the design process through evaluations (Wong, Khong, & Thwaites Reference Wong, Khong and Thwaites2012; Lei et al. Reference Lei, Faust, Rosen and Sherkat2018; Chen, Mata, & Fadel Reference Chen, Mata and Fadel2020).

• • The emotional design approach is linked to the emotional evaluation of a product or system (Norman Reference Norman2004; Bagnara & Smith Reference Bagnara and Smith2006; Agarwal & Meyer Reference Agarwal and Meyer2009).

As for key elements and for the different kinds of experiences, the boundaries between these approaches are not well defined. However, each approach presents specific characteristics. By studying each approach separately, the risk is to limit research on some aspects only, while other elements may be poorly investigated. Otherwise said, if some studies analyse UX only through an emotional approach, the research could be incomplete lacking evidence on the functional and ergonomic side of the HPI (hereinafter, issue comprehensiveness).

2.2. Practical or operational issues

The unclarity characterizing the theoretical research on UX also affects those contributions that investigate UX from a practical perspective. Indeed, when it comes to applying UX within the design process, other issues emerge. Few contributions investigate the involvement of UX in the design process for products belonging to engineering design and everyday items. Such products are characterised mostly by tangible components users can directly interact with and manipulate (Berni & Borgianni Reference Berni and Borgianni2021c). Although UX has a wide range of applications, it seems that experimental design research on UX is limited to the fields of

• • automotive (Williams, Attridge, & Pitts Reference Williams, Attridge and Pitts2011; Lawson, Salanitri, & Waterfield Reference Lawson, Salanitri and Waterfield2016; Shi, Bordegoni, & Caruso Reference Shi, Bordegoni and Caruso2020),

• • healthcare (Harte et al. Reference Harte, Glynn, Rodríguez-Molinero, Baker, Scharf, Quinlan and ÓLaighin2017; McCarthy, Ramírez, & Robinson Reference McCarthy, Ramírez and Robinson2017; Taylor, McDonagh, & Hansen Reference Taylor, McDonagh and Hansen2017)

• • and design for spaces (Brown et al. Reference Brown, Cole, Robinson and Dowlatabadi2010; Jung et al. Reference Jung, Cho, Kim and Kim2018; Ergan et al. Reference Ergan, Radwan, Zou, Tseng and Han2019).

Conversely, most of the experimental activity available in the literature is limited to examining UX in the field of HCI (Bargas-Avila & Hornbæk Reference Bargas-Avila and Hornbæk2012; Córdoba-Cely & Alatriste-Martínez Reference Córdoba-Cely and Alatriste-Martínez2013). Products belonging to this field are indeed electronic devices and ICT products, which are featured by two specific components: software (or digital and intangible) and hardware, related to the physical and tangible parts of the devices (Berni & Borgianni Reference Berni and Borgianni2021c). The reason of such an application field is justified by the origin of the term ‘UX’ itself popularised by Norman, Miller, & Henderson (Reference Norman, Miller and Henderson1995), who gave rise to UX research in HCI. However, the investigation has been mainly restricted to HPI and, markedly, utilisation, especially in electronic and ICT devices. This reduces the analysis of the experience to a very limited and overemphasised set of dimensions such as usability and affordances, neglecting other important dimensions such as emotional processing (Pucillo & Cascini Reference Pucillo and Cascini2014).

Such a strict focus raised other two issues: the intangible components (software) such as services, web app and digital interfaces are usually more investigated than the tangible ones (hardware) (Bargas-Avila & Hornbæk Reference Bargas-Avila and Hornbæk2012). This leads to a lack of generalisability of methods to study UX, limiting the validity of findings and the applicability of methods to many artefacts. Such predominance could be due to the chance to collect objective data straightforwardly in web-based and digital systems and through the opportunities enabled by the Internet of Things (Voet et al. Reference Voet, Altenhof, Ellerich, Schmitt and Linke2019), which is not clearly feasible in the predominant number of consumer goods.

A second issue descends from the popularity of UX in HCI. UX studies often involve advanced technologies and systems such as VR or AR. However, many of these studies aim at investigating the performance of such technologies (e.g., sense of presence and interactivity) rather than focusing on the represented product itself (Gaffary et al. Reference Gaffary, Gouis, Marchal, Argelaguet, Arnaldi and Lécuyer2017; Eroglu et al. Reference Eroglu, Gebhardt, Schmitz, Rausch and Kuhlen2018; Kwon et al. Reference Kwon, Ryu, Ko, Lee, Suh and Nam2019; Shor et al. Reference Shor, Zaaijer, Ahsmann, Weetzel, Immerzeel, Eikelenboom, Hartcher-O’Brien and Aschenbrenner2019; Kim et al. Reference Kim, Kim, Jeong and Kim2020) (hereinafter, issue instruments).

The focus on the interaction with those technologies and their evaluation may lead designers and researchers to lose sight of the applicability of the outputs obtained in these studies. Sometimes, it is difficult to understand exactly what stage of the design process these outputs can be more effectively applied to. Some scholars do not specify which design phase would benefit the most from the application of such objectives. Otherwise, if the utility of the research output is mentioned, it is often referred to the design process in general. In other words, it is difficult to clearly extrapolate design objectives of the contributions (hereinafter, issue clarity of design objectives).

The literature offers a contrasting view on UX data collection. It is unclear whether a more objective or subjective data collection approach to measure UX should be preferred. According to Norman (Reference Norman2004), UX is characterised by emotional components specific to human beings and their feelings. This would make it difficult to measure UX through data collected more objectively (Chou Reference Chou2018). Sutcliffe (Reference Sutcliffe2009) prefers pragmatic approaches focusing just on aspects such as usability and efficiency, which are considered easier to address objectively. The use of more sophisticated procedures and tools, markedly biometric and neurophysiologic devices, has emerged, but established routines and paradigms to investigate design evaluation and product experience are far from being fine-tuned according to a recent review (Borgianni & Maccioni Reference Borgianni and Maccioni2020). It can be observed that design studies use biometric tools to extract information about people’s non-conscious behaviours, but this information is frequently related to specific domains of experience (emotions, use, cognition, affordances) without any example targeting UX as a comprehensive phenomenon. In contrast, other scholars believe that a mixed data collection approach is preferable where subjective data collection methods such as questionnaires, interviews and Likert ratings are placed alongside objective methods (Bitkina, Kim, & Park Reference Bitkina, Kim and Park2020) (hereinafter, issue subjectivity).

The lack of clarity in data collection methods could lead to the development of UX methodologies, approaches and protocols for their own sake. In other words, scholars risk focusing more on the construction of the methodology to evaluate UX, neglecting the potential application of the UX outputs within the design process (hereinafter, issue method demonstration).

Experiments and case studies are a means to validate new methodologies to study UX and demonstrate its operational usefulness in the different stages of the design process. In order for the experimental results to be generalisable, experiments need to have a good ecological validity. In this regard, the selection of the participants for the experiments deserves much attention. While considering participants as a necessary aspect of empirical studies, insufficient attention has been paid to samples. Generally, studies may recruit either experts or novices, depending on whether the research entails, for example, usability testing or product appreciation. Here, experts are generally selected based on their knowledge/experience and they diffusedly include ‘insiders’, for example, people participating in the development of the products to be evaluated. Conversely, even if the estimation of the sample size has a long tradition (Lewis Reference Lewis1994; Schmettow Reference Schmettow2012), the recruitment of novices and laypersons (often people who cannot afford the respective products or would not use them in real life) is far from being representative of any population. In studies evaluating UX, especially in design, these participants are primarily recruited among students or a restricted set of users that often share many features. Moreover, the characterisation of participants is generally made through a set of questions to determine their attitude, while the sample composition is seldom compared to the population of potential users. This lack constrains generalisation and hinders any possibility to explain data further. This might be a reason behind the poor operational benefit of UX studies in terms of steering subsequent design phases (Pucillo & Cascini Reference Pucillo and Cascini2014) (hereinafter, issue participants).

Yet, as long as design phases are concerned, UX studies in design focus mainly on early and final stages of the process. Poor attention is paid to the intermediate stages of prototyping and concept testing. Indeed, UX is used a priori to identify user’s needs, preferences, emotions for idea generation, brainstorming and concept ideation or a posteriori at the end of the design process when the concept is fully developed, and core design decisions cannot be reversed anymore. Hayat et al. (Reference Hayat, Ali, Karaaslan, Cao and Shah2020) claim the importance of UX in early stages. Users’ evaluations are often used for idea generation and brainstorming. Users are usually involved in brainstorming sessions to understand their needs and preferences in terms of design decisions and product features. In other contributions, UX is used at the end of the process when the product is almost fully developed. Users’ needs, preferences and emotions (UX outputs) are identified for concept creation, but it is seldom tested whether the final concept (design output) really reflects the tastes, preferences and needs of users as potential buyers (Ning, Goodman-Deane, & Clarkson Reference Ning, Goodman-Deane and Clarkson2019). This suggests that the intermediate design stages concerning prototyping need to be further investigated.

The exploitation of UX outputs could help reduce the gap between design intentions and user perception leading to the development of satisfactory products requiring less time and investments. The mismatch between designers’ intentions and what users perceive is emphasised by several authors (Crilly, Maier, & Clarkson Reference Crilly, Maier and Clarkson2008; Crilly Reference Crilly2011). In particular, Ning, Goodman-Deane, & Clarkson (Reference Ning, Goodman-Deane and Clarkson2019) underline how the HPI translates into a cognitive challenge to understand the design decisions of a product. In fact, designers rely on their experience, intuition and common sense to make design decisions. According to da Silva, Crilly, & Hekkert (Reference da Silva, Crilly and Hekkert2015), users’ knowledge of the design intentions could also influence their appreciation of a product. The scholars also point out that research has not empirically addressed the problem to whether and how such knowledge can influence the evaluation of a product.

2.3. Main issues in UX in design

Based on the above analysis of the literature, the identified issues have been grouped into eight categories. Table 1 summarises them as follows.

• • The first column indicates the name of the identified issue category, as above.

• • In the second column, an illustration of the issue is provided concisely.

• • In the third column, the criteria used to detect the identified issue category in selected research papers (see Section 3) are reported. In some cases, there can be more than one identification criterion distinguished by the capital letters ‘A’, ‘B’ or ‘C’

Table 1. List of the issues identified in the literature on UX in design

4.1. Overview of the results in quantitative terms

Table 3 presents the count of each ‘Y’, ‘N’ and ‘n/a’ per each category and subcategory of issues. The last column reports the total count of presences or absences of issues.

Table 3. Analysis and count of the number of ‘Y’, ‘N’ and ‘n/a’ per each category of issues

No paper has just ‘Y’ or ‘N’. These data indicate that the categories of issues identified from the literature are validated and present to some extent in all the contributions. However, the prevalence of ‘N’ (Y = 292; N = 580) shows that empirical studies investigating UX in design are overall valid despite the inevitable presence of some limitations. More in details:

• • Despite the prevalence of ‘absence of issue’, there are some categories where the number of ‘Y’ is close to the ‘N’. This is the case of ‘terminology’ (Y = 28; N = 35), where the large number of ‘Y’ suggests that there is still a fair amount of indefiniteness about framing the UX concept at a theoretical level. In particular, the boundaries between usability, UX, its key components and dimensions are often unclear. This frequently leads to consider the UX not as a single concept but as a part belonging to a broader concept.

• • It is worth noting that the prevalence of ‘N’ in ‘method demonstration’ suggests that many studies mention the benefits of their research outputs in design. However, the high number of ‘Y’ in ‘clarity of design objectives A’ indicates that the utility of those outputs within the design process phases is seldom clarified(Yeh, Gregory, & Ritter Reference Yeh, Gregory and Ritter2010; Williams, Attridge, & Pitts Reference Williams, Attridge and Pitts2011; Gkouskos, Normark, & Lundgren Reference Gkouskos, Normark and Lundgren2014; da Silva, Crilly, & Hekkert Reference da Silva, Crilly and Hekkert2015; Zuo et al. Reference Zuo, Jones, Hope and Jones2016; Yoon, Pohlmeyer, & Desmet Reference Yoon, Pohlmeyer and Desmet2017; Zhou et al. Reference Zhou, Lei, Liu and Jiao2017; Zhang et al. Reference Zhang, Wu, Zhang, Wolterink and Xing2018; Alonso-García et al. Reference Alonso-García, Pardo-Vicente, Rodríguez-Parada and Moreno Nieto2020).

• • The large number of ‘n/a’ in ‘participants’ means that, often, data are not acquired directly from potential end users. Other data acquisition techniques have been implemented:

  • ○ online data mining (Chien et al. Reference Chien, Kerh, Lin and Yu2016; Ghajargar et al. Reference Ghajargar, Longo, Gargiulo and Giannantonio2017; Yang et al. Reference Yang, Liu, Liang and Tang2019)

  • ○ user information processed through artificial intelligence (Koonsanit & Nishiuchi Reference Koonsanit and Nishiuchi2021).

In other cases, (see the ‘Y’ in ‘participants B’), UX is studied not only through the interaction and evaluation of end users but also through expert evaluation or analysis of concept case studies where authors comment on the potential interaction between the presented product and the user (Bordegoni et al. Reference Bordegoni, Cugini, Ferrise and Graziosi2014; van der Bijl-Brouwer & van der Voort Reference van der Bijl-Brouwer and van der Voort2014; Bongard-Blanchy et al. Reference Bongard-Blanchy, Bouchard, Bonnardel, Lockner and Aoussat2015; Karana, Barati, & Rognoli Reference Karana, Barati and Rognoli2015; Yoon, Pohlmeyer, & Desmet Reference Yoon, Pohlmeyer and Desmet2017).

• • The conspicuous presence of issues of ‘subjectivity’ indicates that mixed methods of data acquisition are still underutilised.

4.2. Strengths and weaknesses of UX studies exhibiting few issues

The majority of ‘N’ allowed the identification of potential best practices that could be used as a target to conduct empirical UX studies. Following the classification in Table 2, the authors arbitrarily included as ‘best practice’ all the papers (see Table 4) presenting maximum three issues (Y) and two ‘n/a’ designations. These quantities are chosen so to provide for a sufficiently rich description of possible references for future studies.

Table 4. Identification of contributions presenting a possible best practice, that is characterised by a maximum of three ‘Y’ and two ‘n.a.’ and the vast majority of ‘N’

By means of Table 4, readers can notice that the designated best practice examples still suffer from two major issues: ‘participants’ and ‘subjectivity’. These are diffused also if the whole sample of analysed papers is considered (see Table 3). Conversely, the issues ‘methods demonstration’ and ‘clarity of design objectives’ have (almost) no entry despite their relative frequency in the whole sample. Details follow.

The contributions listed in Table 4 can be considered as best practices because they oftentimes present advantages over other studies included in the sample:

• ○ clarity in defining UX since 8 studies out of 10 do not present the ‘terminology’ issue; as a consequence, the presented findings can be actually considered representative of the role of UX in design; hence, results are comparable because they make reference to a solid theoretical construct instead of vague ‘approximations’ of UX;

• ○ in these contributions, neither is the focus on the test of new methodologies, nor are objectives vague (see the issues ‘methods demonstration’ and ‘clarity of design objectives’). These two aspects are possibly logically connected. The ‘best practice’ studies clearly put in practice past knowledge and allow the visualisation of possible applications of UX in design. This is strengthened by their diffused use of a well-defined conceptualisation of UX (see above). Altogether, they are characterised by a superior methodological approach.

Residual issues are supposedly related to contingent factors. ‘Participants’ is the second most frequent category of issues; despite the sample size and characteristics could depend on the objectives of the research, this problem can no longer be neglected as it is increasingly considered as a strong limitation of the validity of design studies (Cash et al. Reference Cash, Isaksson, Maier and Summers2022). This clearly applies to the use of UX in design too. Those studies in which the sampling of participants has been overlooked or has been made based on convenience should clearly address this limitation. In some circumstances, it could be advisable to repeat existing studies to increase the validity of or disconfirm previously achieved results (Cross Reference Cross2018) rather than running research of poor future impact. Moreover, almost all the selected best practices suffer from the issue related to subjectivity. This takes place despite the literature suggests using mixed methods for data acquisition to analyse both quantitative and qualitative data. Here, it is worth noting that the acquisition of objective data takes place through tools and instruments that have made inroads in design only recently (Borgianni & Maccioni Reference Borgianni and Maccioni2020). On the one hand, the papers designated as best practices are, surprisingly, relatively old – just 2 out of 10 have been published in the last 5 years. On the other hand, the presence of new technologies lends itself to focus on their usability and usefulness rather than the outcomes in terms of UX in design. This might explain the contextual absence of ‘method demonstration’ issues and the simultaneous presence of the ‘subjectivity’ issue.

4.3. Comprehensiveness as a proxy of the holistic nature of UX in design

As regards ‘comprehensiveness’, the key components (user, system and context) and dimensions (affective, cognitive and ergonomic) of UX have been counted following the criteria listed in Section 3. In Table 5, the number is summarised of papers that made UX key components (columns) and UX dimensions (rows) explicit. More elements were attributed when multiple key components and dimensions were made explicit in the articles. Table 5 also provides an overview of the different combinations of key components and dimensions. The number at the intersection of columns and rows represents the number of articles analysing those combinations. Under each column and row, there is the total number of each key component and dimension.

Table 5. Analysis and count of the ‘comprehensiveness’ issue. The number refers to the studies considering key factors (user, system, context) and UX dimensions (affective, cognitive, ergonomic)

The total number of dimensions is rather balanced, where ‘cognitive’ results the least considered, while ‘ergonomic’ is the most frequent. Many papers consider two dimensions at a time, but only one studies all the dimensions at the same time (Zuo et al. Reference Zuo, Jones, Hope and Jones2016).

On the other hand, the score for each key factor is not as balanced as for the dimensions. Indeed, ‘system’ emerges as the most considered key factor, while the least considered is ‘context’. Many papers study more than one key factor at once, but only two consider all the key factors at the same time (Hoyos-Ruiz et al. Reference Hoyos-Ruiz, Martínez-Cadavid, Osorio-Gómez and Mejía-Gutiérrez2017; Yang et al. Reference Yang, Liu, Liang and Tang2019).

The same imbalance appears when it comes to different combinations of key factor dimension. The least frequent is ‘context cognitive’ followed by ‘context affective’. Another unusual combination is ‘user cognitive’, while unsurprisingly the system is generally studied in combination with all the UX dimensions homogenously.

Among all the papers, Hoyos-Ruiz et al. (Reference Hoyos-Ruiz, Martínez-Cadavid, Osorio-Gómez and Mejía-Gutiérrez2017) are the closest to provide a holistic view of UX in design. The scholars consider all the key factors and two of the three dimensions. No paper considers all the three key components combined with all the three dimensions contextually.

To summarise, all the key factors and dimensions of UX are overall considered in contributions investigating UX in design. However, some key factors correlated with specific experiences are considered more frequently, as in the case of ‘system cognitive’ and ‘system ergonomic’, while other combinations are still mainly overlooked (‘context cognitive’ and ‘context affective’). Since ‘context’ is the most neglected key factor, future experimental UX studies should investigate the context insightfully in a cognitive perspective. Despite context has been partially overlooked, it has proven to be a crucial factor in studying products’ perceived value (Boztepe Reference Boztepe2007), and judgements (Becattini et al. Reference Becattini, Borgianni, Cascini and Rotini2020), and more knowledge might give rise to effective means to design experiential contexts along with products (Petermans, Wim, & Van Koenraad Reference Petermans, Wim and Van Koenraad2013).

5.1. Are UX outputs considered as an end goal in design only?

The authors have not assigned ranks, priorities and importance to the listed issues. It can be nevertheless remarked that this paper stems from the observed troublesome application of the UX concept in the design process. Otherwise said, it has been questioned whether the leveraging of UX in design is an objective per se or it can be considered as a tool to enhance what is designed. In this respect, some studies are worth highlighting that show a good experimental activity and present the possible benefits of UX within the design process clearly.

Some of those studies are more oriented to improve the design process through early user evaluations or co-working sessions where end users are not only directly involved for user requirements detection but play a role also during concept development. This is the case of Sleeswijk Visser & Visser (Reference Sleeswijk Visser and Visser2006), where potential users served first to detect needs and were later employed in future-scenario techniques to perform and design the interaction between future users and the designed system. The scholars suggest also to ‘re-use’ the same sample of participants in different phases of an iterative design process. Pettersson (Reference Pettersson2017) used a similar ‘future-scenario-acting’ technique to analyse user expectations, understanding their characteristics and highlighting their changes over time. During their study, participants reflected and evaluated the concepts of future cars through enactment and drawing. Another good example of co-design between designers and potential users is provided by Chamorro-Koc, Popovic, & Emmison (Reference Chamorro-Koc, Popovic and Emmison2008), whose research outputs are summarised in design principles to guide the design of product usability. The scholars focused particularly on the forms of representation of a product concept. Yet, as for representation forms and technologic support media, the study of Ferrise et al. (Reference Ferrise, Graziosi, Furtado, Bordegoni and Bongini2013b) is an example of employment of VR and mixed prototyping technology for multimodal interaction. In this study, user evaluation provided feedback for adjusting the flows of the product instantaneously. De Crescenzio et al. (Reference De Crescenzio, Bagassi, Asfaux and Lawson2019) used VR too for concept evaluation and preferences detection to evaluate large-sized products such as aircraft cabin interiors at early stages of the process. Through VR, the scholars simplified the evaluation and the re-design process avoiding time and costs for creating physical prototypes.

Other studies involve users mainly to detect previously undisclosed custom requirements. Belz (Reference Belz2006) used UX outputs from the interaction between customers and a digital camera to find potentialities, flows and new users’ needs, preferences and behaviours to improve their product. More recently, McCarthy, Ramírez, & Robinson (Reference McCarthy, Ramírez and Robinson2017) had the similar aim to identify users’ requirements, behaviour and preferences, but they studied UX from a more emotion-oriented perspective in the field of healthcare design. The identification of young users’ requirements while using a medical device for diabetes monitoring served designers to create several new device concepts. Each concept adapted to the different contexts and use situations highlighted by young users during preference detection. Lin et al. (Reference Lin, Chen, Hsiao and Lin2016) considered the context as a factor to be implemented in the design of products as well. The scholars focused on the analysis of the ‘cultural’ meanings of the context. The scholars used a specific cultural object to raise awareness of cultural pattern and identify hidden cultural meanings to be used as input to develop new concepts. Their study showed how it is possible to include cultural features into new products that can appeal young consumers.

All these examples show how UX can improve the product during its development through the detection of users’ needs and preferences. Some studies focus more on the iterative nature of the design process and therefore, their research output has been used to improve the design phases through iterations. This is highlighted in Torres et al. (Reference Torres, Heck, Rudd and Kelley2009), who present a case study to prove best practices for UCD in the military field. They developed a taxonomy whose information can be leveraged throughout the entire system to guide an end user in the maze of complex tasks during evaluation and prototype interaction. Liikkanen & Reavey (Reference Liikkanen and Reavey2015) used ‘resonance testing’ to collect evaluation feedback and revise the design concept iteratively during its development. Another example is the work made by Yoon, Kim, & Kang (Reference Yoon, Kim and Kang2020). The scholars used retrospective techniques to analyse the dynamic nature of UX over time. This paper reveals how the patterns of positive UX in relation to a product vary over the usage life cycle, from before purchase to disposal/repurchase. Keeping in mind such information can be of paramount importance for designers to optimise the iterative process of the development of new products.

Overall, studying UX in design is not effortless, especially when the aim is to achieve scientific soundness. It is not easy to avail of a sufficient number of participants to form a representative population, collect both objective and subjective data though mixed methods and use the UX outputs to establish the next steps in the design process form. However, the contributions above can be used as valuable best practices despite some limitations.

5.2. Issues as a checklist for the planning of future UX-based design studies

Although the presented list of issues cannot be considered comprehensive, the highlighted issues have been justified both in a theoretical (through the scoping review of critical aspects in UX) and a practical perspective (through the systematic search of UX applications in design). These verified issues lead to the implication that upcoming studies can be designed in a way to minimise the number of known issues and their impact on the validity of research.

Otherwise said, future research involving the application of UX in design can benefit from the paper’s findings and use the issues as a checklist for designing experiments accurately. The process the authors propose to plan UX experiments in design is illustrated through the flow diagram made available in the Supplementary Materials a pdf document. The flow diagram guides a researcher through the evaluation of their current design of experiments and helps fix possible shortcomings by addressing all issues and identification criteria included in Table 1, as well as report insurmountable limitations due to peculiar contingency factors. The documentation of what of UX is actually studied, how this is done and the residual limitations allows future studies of UX in design to be effectively compared. In detail, the structure of the flow diagram sequentially foresees:

• • The definition of what is intended for UX, the designation of the willingness or need to address UX in a holistic way and the determination of the studied domains (issues ‘terminology’ and ‘comprehensiveness’);

• • The declaration of intended objectives and expected results of UX-based design studies (issue ‘method demonstration’);

• • The fine-tuning of the experiment through the sampling of participants, the use of required technologies, the setting and timing of interactions, the employment of the needed tools to extract the intended data (issues ‘instruments’, ‘participants’, ‘retrospective/projected’ and ‘subjectivity’);

• • The reporting of the experiment and how the design of the involved products is affected by the UX-based study and the general implications for design research (issue ‘clarity of design objectives’).

Obviously, the flow diagram is intended as a dynamic representation of recommendations to follow for a proper experimental UX-based study in design and the identification of additional issues can lead to its reformulation. The sequence of presented issues is proposed by the authors, but scholars might be willing to follow a difference logical order.