2.1. Systematic mapping study

The choice of methodology for the review of trustworthy AI principles was determined by the fact that, despite increased recent research around trustworthy, responsible, and ethical AI, this particular research area is still in a nascent stage, compared, for instance, to the more than seven decades of research in the wider field of AI. Hence, a typical systematic literature review would not be appropriate as the aim is not to aggregate evidence on how trustworthy, responsible, and AI practices have been achieved, when such evidence is still not plentiful. In contrast, a systematic mapping study is designed to specifically understand the structure of a research area and the principles that underlie it, rather than gathering and synthesizing evidence (Petersen et al., Reference Petersen, Vakkalanka and Kuzniarz2015). As such, it is the appropriate tool to understand and map the range of principles that are driving the development of trustworthy, responsible, and ethical AI solutions.

In the context of software engineering and computer science, an established methodology for conducting systematic mapping studies is that of Petersen et al. (Reference Petersen, Feldt, Mujtaba and Mattsson2008) (updated in Petersen et al., Reference Petersen, Vakkalanka and Kuzniarz2015). While the methodology follows the general structure of systematic mapping studies as outlined by Petersen et al. (Reference Petersen, Vakkalanka and Kuzniarz2015), our focus was on synthesizing relevant literature thematically in response to the three guiding research questions, rather than on performing a quantitative mapping of publication trends or keyword frequencies. In the remainder of this section, we outline how we applied this methodology for our exploration of trustworthy AI principles.

The first step in the systematic mapping study process is to define research questions. The following research questions guide our mapping study:

• • What are the main concepts of responsibility, ethics, and trustworthiness and their interrelationships?

• • What are the different concerns arising in relation to algorithmic ethics?

• • How do trustworthy AI principles apply to development practices?

The search strategy chosen for the mapping study was an automated search on online databases. As is typical for mapping studies, we used the research questions as a basis for deciding on keywords and opted for a two-level structure, with the two levels conjunctively combined with AND. The first level includes the main focus of the study, AI, and includes the disjunction “artificial intelligence” OR AI. The second level includes the following disjunction of terms related to trustworthy AI: “trustworthy OR responsible OR ethical.” Note that we opted for “AI” as a general keyword, rather than specific keywords referring to specific AI algorithms, as this reflects the terminology used by the majority of researchers and practitioners in manufacturing and supply chain disciplines, who frequently adopt the general label ‘AI’ even when referring to specific subfields.

To maximize coverage, we conducted our search across three major online databases: Scopus, IEEE Xplore, and ACM Digital Library. We chose Scopus over Web of Science based on literature that confirms it has a slightly wider coverage (Pranckuté, Reference Pranckuté2021). While Scopus does include the majority of IEEE and ACM publications, it is not guaranteed to include all of these, so we chose to search these publishers’ databases as well. In terms of time period, we focused on publications from 2016 until today, given that the attention to trustworthy AI is a relatively recent development. The latest searches, the results of which are presented in Section 3 were conducted on 17 February 2025.

The following inclusion criteria were applied: Studies must be written in English and published in journals. Additionally, publications must relate to one or more of the defined research questions by offering a conceptualization of trustworthy AI from a theoretical or practical perspective. In terms of exclusion criteria, we opted not to exclude sources that are not specifically related to manufacturing and supply chains, as this would significantly limit coverage given that trustworthy AI, and AI in general are comparatively less researched in these domains compared to other domains such as healthcare; instead, we provide a brief commentary on the relevance of trustworthy AI to manufacturing in Section 3.5, while the second part of this paper focuses on exploring trustworthy AI challenges that are illustrated through examples within the manufacturing and supply chains domain.

Our search resulted in 3858 articles from Scopus, 238 from IEEE, and 127 from ACM. Following the search, filtering based on inclusion and exclusion criteria was conducted, followed by full-text reading was conducted. Filtering resulted in 48 papers retained. An additional 22 papers were added to the study corpus through snowball sampling, resulting in a corpus of 71 papers. Finally, data extraction and analysis of reviewed publications were conducted independently by two of the co-authors, followed by a consensus meeting, as is common in most mapping studies in the literature (Petersen et al., Reference Petersen, Vakkalanka and Kuzniarz2015). Both the search and its results, as well as the mapping study outputs, were reviewed by the last co-author, as a key researcher in trust, compliance, and accountability of emerging and data-driven technologies. Figure 1 presents a flow diagram for the process followed.

Figure 1. Flow diagram for the systematic mapping study.

Results of the mapping study conducted using the above-described methodology are presented in Section 3, with Sections 3.2, 3.3 and 3.4, respectively, focusing on the first, second, and third research question and Section 3.5 summarizing and relating the study results to manufacturing.

2.2. Synthesis and research agenda

For the second part of our study, we designed a bespoke methodology to structure our analysis and systematically identify the different sources of potential harm to the development of trustworthy AI in the manufacturing and supply chain domain. This methodology is grounded on two main principles: alignment with well-established AI life cycles to ensure complete coverage of all aspects and case study-based analysis to establish links between trustworthy AI development challenges and tangible potential impact on manufacturing and supply chain operations.

The proposed methodology uses an adapted version of the machine learning (ML) life cycle definition suggested by Ashmore et al. (Reference Ashmore, Calinescu and Paterson2021). As discussed by Suresh and Guttag (Reference Suresh and Guttag2021), sources of undesirable behavior are not just due to “biased data.” The ML pipeline involves a series of choices and practices, including but not limited to model creation, training, and verification, that can lead to undesirable effects. Using Ashmore’s life cycle as the basis for our methodology allows us to systematically identify the sources of potential challenges and what they mean within a manufacturing and supply chain context. For each phase in the life cycle and each activity within each phase, we explore a wide variety of issues that relate to trustworthy AI. In addition to the synthesis of trustworthy AI challenges, which is presented in Section 4, we go beyond individual phases and activities and explore cross-cutting considerations as well, in Section 5.

As far as ensuring that our exploration of trustworthy AI issues is linked to their tangible impact on manufacturing and supply chain operations, we opted to make use of illustrative examples that explain how each issue raised may result in a given trustworthiness concern. Due to the limited research and development of trustworthy AI solutions in the manufacturing and supply chain domains, the provided examples are drawn from expert knowledge and industry experience.

Illustrative examples are also then coupled with one or more research questions that identify potential areas for research to address the challenges identified and mitigate their impact. Research questions were derived through abductive reasoning informed by illustrative examples of real-world industrial practice encountered through the authors’ ongoing engagements with AI deployment in manufacturing and supply chain contexts. These research questions arise from challenges encountered in articulating and applying ethical and trustworthy AI principles across different industrial stages, from initial scoping and design, to procurement, implementation, deployment, and assurance. They also reflect recurring concerns around bias, fairness, explainability, accountability, risk management, and regulatory uncertainty. The questions were designed to reflect the recurring themes observed in these engagements, while remaining broad enough to allow mapping to the wider academic and policy literature. This structure enables the study to bridge theoretical formulations with practical concerns emerging in applied settings. Collectively, these research questions constitute our proposed research agenda for future exploration of trustworthy AI in the context of manufacturing and supply chains. A summary of the results of the second part of our study is presented in Table 2.

3.1. Bibliometric analysis

To provide a broader overview of the trustworthy AI literature, we supplemented the systematic mapping study with a bibliometric analysis using VOSviewer (van Eck and Waltman, Reference van Eck and Waltman2010). This analysis aimed to explore recurring themes, geographical research activity, and patterns of scholarly collaboration across the included corpus. Four visualizations were generated: one based on keyword co-occurrence (Figure 2, one on country-level co-authorship (Figure 3), and two on author collaboration networks (Figure 4).

Figure 2. Keyword co-occurrence network visualizing the conceptual structure of trustworthy/responsible AI literature. Node size reflects term frequency; colours indicate thematic clusters.

Figure 3. Geographical distribution of research contributions to the trustworthy AI literature. Node size indicates publication volume; proximity indicates collaboration strength.

Figure 4. Author co-authorship networks in the trustworthy AI literature.

The keyword co-occurrence map (Figure 2) offers an overview of the main topics addressed in the literature. The central position of terms such as artificial intelligence, machine learning, and ethics reflects their foundational role in this domain. Around these core concepts, several distinct clusters are visible. For instance, terms like explainability, accountability, and fairness form a tightly connected group that corresponds to ongoing discussions around ethical principles in AI. Other terms, such as governance, human–computer interaction, and philosophical aspects, indicate the field’s interdisciplinarity and engagement with policy and sociotechnical issues. The visualization highlights the prominence of technical concepts such as explainable AI alongside more normative concerns, underscoring the hybrid nature of current research in this area.

Figure 3 illustrates the geographical distribution of authorship and co-authorship. The most active countries in terms of publication volume and collaboration include the United Kingdom, the United States, Germany, and Italy. A relatively dense cluster of European countries is visible, suggesting strong regional collaboration, while countries such as China, Australia, and Canada appear as active contributors with varying degrees of international co-authorship. Several countries from the Global South, including Nigeria and Egypt, are represented but relatively disconnected. This visualization draws attention to the asymmetries in global participation and highlights the importance of ensuring broader inclusion in international AI governance debates.

Figure 4a and b present author-level co-authorship networks. The first (Figure 4a) shows the largest connected component, centered on a group of researchers who frequently collaborate on issues such as algorithmic bias, transparency, and AI governance. The second (Figure 4b) extends this view to include all clusters with link strengths greater than 1. This wider perspective reveals the fragmentation of the field into several relatively distinct communities, each with its own internal collaborations but limited inter-group connectivity. The lack of cross-cluster ties suggests that, although the field is active and collaborative, there may be missed opportunities for interdisciplinary integration—particularly between technical, legal, and social science perspectives on trustworthy AI.

3.2. Relationships among responsibility, ethics, and trustworthiness

With regard to the first research question of the mapping study on concepts of responsibility, ethics, and trustworthiness, one common approach in the literature, which we follow in our analysis, is to view responsible and ethical requirements as being fundamental prerequisites to trusting an AI system. According to Smuha (Reference Smuha2019), the European Union defines trustworthy AI as being “lawful (respecting all applicable laws and regulations), ethical (respecting ethical principles and values) and robust (both from a technical perspective while taking into account its social environment).” Responsibility in the form of accountability is defined as one of seven key requirements that AI systems should meet in order to be deemed trustworthy, encompassing responsible development, deployment, and use.

In his analysis of reliable, safe, and trustworthy AI, Shneiderman (Reference Shneiderman2020) primarily views responsibility from the perspective of clarifying the role of humans in AI failures, also mentioning responsibility in combination with fairness and explainability as goals of human-centred AI. Kaur et al. (Reference Kaur, Uslu, Rittichier and Durresi2022) defines accountability/responsibility as one of the requirements for trustworthy AI, alongside fairness, explainability, privacy, and acceptance.

Thiebes et al. (Reference Thiebes, Lins and Sunyaev2021) posits that “AI is perceived as trustworthy by its users (e.g., consumers, organizations, society) when it is developed, deployed, and used in ways that not only ensure its compliance with all relevant laws and its robustness but especially its adherence to general ethical principles.” For the latter, they adopt the following ethical principles: beneficence, non-maleficence, autonomy, justice, and explicability. Responsibility is only considered as an aspect of explicability and justice, in the sense of holding someone legally responsible in case of an AI failure.

Trustworthiness is also at the core of the recently published white paper of the UK Government on AI regulation (Department for Science, Innovation and Technology, 2023). One of the three main aims of the proposed regulatory framework is to increase public trust in the use and application of AI and is underpinned by five principles: safety, security and robustness, appropriate transparency and explainability, fairness, accountability and governance, and contestability and redress.

Finally, Newman (Reference Newman2023) also places trustworthiness at the centre of discussions around responsibility and ethics, and develops a comprehensive taxonomy of 150 trustworthiness properties. These properties relate to one of the following eight trustworthiness characteristics based on NIST’s AI Risk Management Framework (Trustworthy and Responsible AI Resource Center, 2023): valid and reliable, safe, secure, and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, fair with harmful bias managed, and responsible practice and use. This taxonomy clearly positions responsibility and ethics as contributors and prerequisites to trustworthiness, rather than outcomes of it.

Our mapping confirms that this view remains prevalent in recent literature. For example, Mentzas et al. (Reference Mentzas, Fikardos, Lepenioti and Apostolou2024) and Kattnig et al. (Reference Kattnig, Angerschmid, Reichel and Kern2024) reaffirm the framing of responsibility and accountability as components of trustworthiness, while authors such as Stahl (Reference Stahl2023) and Law et al. (Reference Law, Ye and Lei2025) explore how the broader ecosystems and societal domains in which AI operates shape the interpretation of responsibility and trust. A number of sources continue to treat ethical principles such as justice and autonomy as foundational to earning trust in AI, especially in domain-specific applications such as healthcare (Zhang and Zhang, Reference Zhang and Zhang2023; Ueda et al., Reference Ueda, Kakinuma, Fujita, Kamagata, Fushimi, Ito, Matsui, Nozaki, Nakaura, Fujima, Tatsugami, Yanagawa, Hirata, Yamada, Tsuboyama, Kawamura, Fujioka and Naganawa2024).

Note, however, that an alternative viewpoint is to cast trustworthiness as a prerequisite for responsible AI. Wang et al. (Reference Wang, Xiong and Olya2020) groups responsible for AI practices into four categories: training/education, risk control, ethical design, and data governance. Trust building is one component of data governance, along with explainability and transparency. The narrative provided is a quite narrow view of trust that is centred around reducing bias through high-quality data and ensuring there is consent for sharing data.

Arrieta et al. (Reference Arrieta, Díaz-Rodríguez, Del Ser, Bennetot, Tabik, Barbado, García, Gil-López, Molina and Benjamins2020) defines seven responsible AI principles: explainability, fairness, privacy, accountability, ethics, transparency, security/safety. Trust is not included independently, but rather shown as an aspect or goal of explainability. As the authors explain, trustworthiness and explainability are not equivalent, as being able to explain outcomes does not imply that they are trustworthy, and vice versa.

Rather than viewing one as a prerequisite of the other, some researchers place both responsibility and trustworthiness at the same level, as principles of ethical AI. A comprehensive review of AI guidelines in the literature is conducted by Jobin et al. (Reference Jobin, Ienca and Vayena2019), producing the following list of ethical AI principles in order of commonality: transparency, justice/fairness, non-maleficence, responsibility /accountability, privacy, beneficence, freedom /autonomy, trust. Responsibility and accountability are rarely defined, but recommendations focus on “acting with integrity and clarifying the attribution of responsibility and legal liability.” Trust is referenced in relation to customers trusting developers and organizations and trustworthy design principles.

3.3. Algorithmic ethics

We now focus our attention on the second research question around “ethical algorithms.” Mittelstadt et al. (Reference Mittelstadt, Allo, Taddeo, Wachter and Floridi2016) developed a map with different types of ethical concerns useful for doing a rigorous diagnosis of ethical concerns emerging from AI, which are used to evaluate ethical outcomes in AI applications. In their map, “inconclusive evidence” refers to the data analysis stage where results produce probabilities but also uncertain knowledge. Here authors point out cases where correlations are identified but the existence of a causal connection cannot be posited. Failure to recognize this might then lead to unjustified actions. “Inscrutable evidence” refers to a lack of transparency regarding both the data used to train an algorithm and a lack of interpretability of how data-points were used by an algorithm to contribute to the conclusion it generates. This is commonly referred to as the “black-box” issue, leading to non-obvious connections between the data used and the resulting conclusions. ‘Misguided evidence’ refers to the fact that an algorithmic output can never exceed the input and thus conclusions can only be as reliable and neutral as the data they are based on, which can lead to biases. ‘Unfair outcomes’ refer to actions that are based on conclusive, scrutable, and well-founded evidence, but they have a disproportionate, disadvantageous impact on one group of people, often leading to discrimination. “Transformative effects” refer to algorithmic activities, such as profiling the world by understanding and conceptualizing it in new, unexpected ways, triggering and motivating actions based on the insights it generates (Morley et al., Reference Morley, Machado, Burr, Cowls, Joshi, Taddeo and Floridi2020). This can lead to challenges for autonomy and informational privacy.

Ethics by design include best practices in the development of AI to mitigate the above group of concerns—for example, the establishment of an ethics board (Leidner and Plachouras, Reference Leidner and Plachouras2017) and integration of “ethical decision routines in AI systems” (Hagendorff, Reference Hagendorff2020), whereby decision algorithms are explicitly designed to respect ethical values.

Many papers in our extended mapping continue to reflect and expand on these concerns. Hagendorff (Reference Hagendorff2020) provides a critical assessment of over 20 ethical AI guidelines, identifying key omissions and inconsistencies in how algorithmic harms are conceptualized. Kazim and Koshiyama (Reference Kazim and Koshiyama2021) and Lewis and Marsh (Reference Lewis and Marsh2022) explore how algorithmic manipulation and opacity undermine human agency and trust, while Wang et al. (Reference Wang, Liu, Yang, Guo, Wu and Liu2023) and Zhang and Zhang (Reference Zhang and Zhang2023) document the emergence of ethical risks in clinical AI systems, such as automation bias and unequal access to treatment.

A number of studies stress the importance of domain context when considering algorithmic ethics. For instance, Giovanola and Tiribelli (Reference Giovanola and Tiribelli2023) argue that standard definitions of fairness are insufficient for healthcare settings and propose a “relational fairness” approach that considers patient dignity and systemic inequalities. Similarly, Nguyen et al. (Reference Nguyen, Ngo, Hong, Dang and Nguyen2023) and Tang et al. (Reference Tang, Li and Fantus2023) examine stakeholder perceptions of fairness and transparency in education and medical AI, respectively, identifying mismatches between technical fairness mechanisms and societal expectations.

Several contributions also examine the socio-technical nature of algorithmic ethics, calling for a shift from principle-level discussions to implementation frameworks. Starke et al. (Reference Starke, Baleis, Keller and Marcinkowski2022) conduct a systematic review of perceived algorithmic fairness and note that technical fixes alone are unlikely to address public concerns. Others, such as Radanliev (Reference Radanliev, Santos, Brandon-Jones and Joinson2024) explores the use of privacy-preserving technologies like federated learning and homomorphic encryption as a means of embedding ethics into algorithmic architecture.

3.4. Developing trustworthy AI

In contrast to the first two areas of the mapping studies, which focused on principles underlying trustworthy AI research, the third area looks at trustworthy AI from the point of view of development. Building on principles, practitioners must consider, undertake and employ various measures and safeguards so as to mitigate the risks of the technology, such that they consider and address the various concerns to avoid negative consequences on human and societal well-being (Dignum, Reference Dignum2023).

Toward this, there have been various approaches describing responsible development practices. For example, Arrieta et al. (Reference Arrieta, Díaz-Rodríguez, Del Ser, Bennetot, Tabik, Barbado, García, Gil-López, Molina and Benjamins2020) and Sambasivan and Holbrook (Reference Sambasivan and Holbrook2018) describe responsible AI as being concerned with the design, implementation and use of ethical, transparent, and accountable AI technology in order to reduce biases, promote fairness, equality, and help facilitate interpretability and explainability of outcomes.

When conceptualizing responsible AI, the principles of responsible research and innovation (RRI) have served as a starting point to consider and anticipate the consequences of a particular technology in society (Owen et al., Reference Owen, Macnaghten and Stilgoe2012). Shaped by contributions from Science and Technology Studies, this approach has been established and prominent in recent projects funded by the European Commission. The application roadmap of responsible AI includes continuous reflection on context and civil society, such as third sector organizations, so as to align the AI development process and outcomes with society’s expectations. Decision processes must be visible and transparent to ensure that developers are on track regarding their responsibilities, and the development process must allow users and stakeholders of technologies to criticize outcomes.

In terms of areas of consideration specific to the AI lifecycle, the topics of transparency (including explainability and interpretability) and fairness (bias) have received considerable attention by the technical and engineering communities, which we explore below. Note, however, that there is a clear realization that issues of trustworthy AI are inherently socio-technical (Kroll et al., Reference Kroll, Huey, Barocas, Felten, Reidenberg, Robinson and Yu2017; Raji et al., Reference Raji, Smart, White, Mitchell, Gebru, Hutchinson, Smith-Loud, Theron and Barnes2020), and require a consideration of technical, organizational, and human processes aspects, throughout the technology development, operation, and use (Cobbe et al., Reference Cobbe, Lee and Singh2021) as well as their supply chains (Cobbe et al., Reference Cobbe, Veale and Singh2023).

Recent literature extends this understanding by introducing development frameworks tailored to specific trustworthiness requirements. For example, Thiebes et al. (Reference Thiebes, Lins and Sunyaev2021) propose the DaRe4TAI framework, which operationalizes five trustworthiness principles (including autonomy and justice) through a series of design dimensions. Similarly, Mentzas et al. (Reference Mentzas, Fikardos, Lepenioti and Apostolou2024) provide a structured overview of methods and toolkits for implementing trustworthy AI in practice, including risk assessment, fairness evaluation, and transparency monitoring.

The use of standardized development guidance is also highlighted in technical and regulatory frameworks. Several papers refer to the NIST AI Risk Management Framework and the assessment list for trustworthy artificial intelligence (ALTAI) as concrete tools to translate ethical principles into engineering practices (Díaz-Rodríguez et al., Reference Díaz-Rodríguez, Del Ser, Coeckelbergh, López de Prado, Herrera-Viedma and Herrera2023; Radclyffe et al., Reference Radclyffe, Ribeiro and Wortham2023; Kattnig et al., Reference Kattnig, Angerschmid, Reichel and Kern2024). These studies suggest that development practices must be embedded across the AI lifecycle, from data preparation to post-deployment auditing.

Some contributions also emphasize the limitations of high-level frameworks in practice. Morley et al. (Reference Morley, Kinsey, Elhalal, Garcia, Ziosi and Floridi2023) find that developers and policymakers often struggle to operationalize principles such as fairness and accountability without domain-specific guidance. Stahl (Reference Stahl2023) calls for a shift toward ecosystem-level responsibility, recognizing that AI trustworthiness depends on the combined actions of system developers, deployers, regulators, and affected communities.

Finally, we find an increasing number of domain-specific development strategies that adapt general trustworthiness principles to localized contexts. For instance, Ueda et al. (Reference Ueda, Kakinuma, Fujita, Kamagata, Fushimi, Ito, Matsui, Nozaki, Nakaura, Fujima, Tatsugami, Yanagawa, Hirata, Yamada, Tsuboyama, Kawamura, Fujioka and Naganawa2024) introduce the FAIR framework for healthcare AI development, and Law et al. (Reference Law, Ye and Lei2025) propose a staged AI evolution trajectory model tailored for the hospitality sector. These approaches illustrate the growing maturity of trustworthy AI discourse, which is moving from abstract discussion to actionable development practices.

3.5. Trustworthy AI principles and their relevance to manufacturing

Table 1 summarizes the results of the mapping study in the form of the most common requirements for trustworthy AI proposed in the literature. Although some authors use some of these terms to encompass other requirements, the taxonomy by Newman (Reference Newman2023) provides a framework that encompasses most aspects in literature.

Table 1. Mapping of literature to Newman’s AI trustworthiness requirements

Note. New additions are shown in blue.

As we see from the above terminology, there is considerable work pertaining to responsible, trustworthy, and ethical AI principles and these often overlap, with debate taking place over their specific taxonomy. Defining such a taxonomy is an important area of deliberation, as it allows for structured thinking. However, for the purposes of this paper, we refrain from strictly defining this fluid field and opt to be as broad and inclusive as possible. For this reason, we use the term “trustworthy AI” as an umbrella term to encompass principles of ethical, responsible, and trustworthy AI.

We argue that manufacturers need to invest in all trustworthy AI principles, and not just a subset of them, when considering how they develop, deploy, and practice AI in their organizations. Figure 5 illustrates how each principle in the taxonomy by Newman (Reference Newman2023) has an impact on different aspects of manufacturing.

Figure 5. Mapping key principles in Newman (Reference Newman2023) to corresponding needs in the manufacturing context.

Manufacturing companies do not exist in isolation. They impact not only the profitability of stakeholders but also the well-being of their workers. The products and services that manufacturing engineers design and produce impact society. The use of natural resources and waste that is generated during production and delivery has a profound impact on the environment, and decisions companies make on suppliers can have wide wide-reaching impact on global economies. Therefore, it is crucial that manufacturing adopts AI in a lawful and ethical manner that is robust, safe, and avoids negative consequences to human society and well-being. It is important to be able to prove that an organization does so, via algorithms and datasets that can be scrutinized. Given its wide-ranging remit, we thus feel that the field of manufacturing needs to be inclusive when thinking about trustworthy AI principles. Thus, in the remainder of this paper, we shall review specific challenges that manufacturing must face in order to be able to create and deploy trustworthy AI in its broadest sense.

4.1. Data management

Data management focuses on the preparation of datasets needed to build an AI model, which typically include data collection, augmentation, and pre-processing processes.

4.2. Model creation

Following the data management part of the AI lifecycle, an AI model is created either through a model learning process, in the case of ML, or through knowledge engineering, in the case of knowledge-based AI. In this section, we focus primarily on challenges affecting model learning. This is because knowledge engineering is a human-centred process; as such, trustworthiness is less likely to be compromised as a result of the model creation process itself and is more a reflection of the trustworthiness of the human experts involved.

4.3. Ethical governance and Interoperability

A growing concern is the privacy of personal data used to train AI, primarily ML-based ones. In a manufacturing scenario, this may involve end-user (customer) data, as well as supplier, business relationship data, and data from employees. The ethical implications of violating worker privacy are a growing concern that crucially needs more attention in the context of manufacturing. Surveillance mechanisms deployed on the factory floor are a prime example (De Cremer and Stollberger, Reference De Cremer and Stollberger2022). Although manufacturing-specific surveys have not yet been conducted, in 2017, a global survey found that over 69% of companies with at least 10,000 employees have an HR analytics team that uses automated technologies to hire, reward, and monitor employees.

In many cases, shopfloor worker monitoring may have been deployed with valid and ethical intentions. These may include, for example, ensuring Personal Protection Equipment (PPE) has been worn correctly, identifying hazards, aiding novice workers with suggestions on how to complete a difficult task, quality certification of products that necessitate an evidence trail that processing steps were performed adequately. However, the same technology can be used in ways that are ultimately detrimental to the well-being of workers, erode human-centered values, and jeopardize individual rights to self-determination.

Unsurprisingly, there are limited cases that have been brought to light, and even fewer academic studies. One of the high-profile cases has been reported by the Open Markets Institute, an advocacy group focusing on technology company monopolies (Hanley and Hubbard, Reference Hanley and Hubbard2020). They found that Amazon uses a combination of tracking software, item scanners, wristbands, thermal cameras, security cameras, and recorded footage to monitor the activities of warehouse workers. Whistle-blowers suggested that workers need to wear an item scanning machine (scan gun) which detects “idle time.” The scan gun alerted a manager if workers spent over the maximum allowance of 18 minutes of idle time per shift. Idle time included bathroom breaks, getting water, or walking slower, and thus could be easily exceeded. The algorithm that powered the scan gun would classify idle time based on expected levels of motion. Two other cases from Amazon included recognizing when a forklift driver has been yawning, which the drivers saw as an invasion of their privacy; and the use of employee personal data in conjunction with shopfloor worker monitoring data to prevent unionization. In cases such as worker monitoring, there is an inherent power imbalance between the employer and employees, making it hard for workers to question data being gathered about them and algorithms used to analyse their data.

Remote working during the Covid-19 pandemic has increased reports of privacy invasion, for example, by software that detects worker productivity through monitoring keyboard strokes. Other commercially available software allows company managers to map company social networks by using email metadata and detecting employee “sentiment” through email conversations, and even predicting when an employee is showing signs of frustration and may want to leave the company. The market for HR Analytics software, which includes manufacturing worker surveillance, is projected to reach USD 11 billion by 2031.

Although worker monitoring itself is not a new concept, in the case of AI the fear is that monitoring can be scaled up by including multiple, often objectionable and private data sources (transformative outcomes), and inference is automated without any real insight to the algorithmic decision process or data itself (inscrutable and inconclusive evidence), potentially resulting in discriminatory practices and undue pressure on employees, effecting their well-being (unfair outcomes). The power imbalance between workers and managers means that workers often have no say or are hesitant to say whether and how such technology should be adopted.

In response, several countries are proposing regulations to prevent ethical issues arising from AI, but more research is needed to inform regulators of the potential consequences of the misuse of algorithms to benefit commercial interests in the manufacturing workplace, and ensure that regulations are effective. The White House Office of Science and Technology issued the Blueprint for an AI Bill of Rights in October 2022 (Office of Science and Technology Policy, 2022), with the aim of protecting civil rights and democratic values in the development and use of automated systems. The Bill of Rights highlights the need for data privacy in the employment context. Again, in the United States (and one province in Canada), three states have implemented laws that require employers to notify employees of electronic monitoring, including AI-powered technologies, by providing notice to employees whose phone calls, emails, or internet usage will be monitored. In New York, automated decision tools that replace or assist in hiring or promotion decision-making must undergo annual bias audits. Companies must make the audit results publicly available and offer an alternative selection process for employees who do not want to be reviewed by such tools.

In Europe, both Norwegian and Portuguese Data Protection Authorities outlawed the practice of remote worker monitoring. The European Union is drafting an Artificial Intelligence Act (EU AI Act [European Commission, 2021]) to regulate AI, in which “employment, management of workers, and access to self-employment” are considered high risk and will be heavily regulated. The EU’s General Data Protection Regulation (GDPR) limited the use of AI in employment, explicitly stating that employees should not be subject to decisions “based solely on automated processing.” The UK has published an AI regulation white paper (Department for Science, Innovation and Technology, 2023). The white paper’s focus is on coordinating existing regulators such as the Competition and Markets Authority and Health and Safety Executive, but it does not propose any regulatory power. Critics raised that the UK’s approach has significant gaps, which could leave harms unaddressed, relative to the urgency and scale of the challenges AI brings (Hern, Reference Hern2023).

The White House Office of Science and Technology issued the Blueprint for an AI Bill of Rights in October 2022, with the aim of protecting civil rights and democratic values in the development and use of automated systems. The Bill of Rights highlights the need for data privacy in the employment context. Again, in the United States (and one province in Canada), three states have implemented laws that require employers to notify employees of electronic monitoring, including AI-powered technologies, by providing notice to employees whose phone calls, emails, or internet usage will be monitored. In New York, automated decision tools that replace or assist in hiring or promotion decision-making must undergo annual bias audits. Companies must make the audit results publicly available and offer an alternative selection process for employees who do not want to be reviewed by such tools. Both Norwegian and Portuguese Data Protection Authorities outlawed the practice of remote worker monitoring. In the EU AI Act, “employment, management of workers, and access to self-employment” are considered high-risk and are heavily regulated. The EU’s General Data Protection Regulation (GDPR) limited the use of AI in employment, explicitly stating that employees should not be subject to decisions “based solely on automated processing.” The UK has published an AI regulation whitepaper (Department for Science, Innovation and Technology, 2023). The whitepaper’s focus is on coordinating existing regulators, such as the Competition and Markets Authority and Health and Safety Executive, but does not propose any regulatory power. Critics raised that the UK’s approach has significant gaps, which could leave harms unaddressed, relative to the urgency and scale of the challenges AI brings (Hern, Reference Hern2023).

While the regulatory debate is encouraging, we note that the effectiveness of these regulatory initiatives in the manufacturing and supply chain sector remains to be seen. At the moment, these frameworks are not enforceable, and progress is not fast enough to keep up with the fast pace of AI research developments. Although multiple high-level frameworks have been created, as summarized in Section 3, there is a lack of practical use cases and guidance on how these frameworks can be incorporated into daily business practice. This is especially true in the field of manufacturing.

At the time of writing, ISO, IEC, and BSI standards are being discussed and proposed (https://aistandardshub.org/) (e.g., ISO/IEC TR 24028:2020). However, at the moment, there is no consensus on the adoption of trustworthy AI standards. Should companies wish to develop surveillance mechanisms on their workers to prevent, for example, unionization, they are free to do so. Participation in industry standards may play an important role in designing effective regulatory frameworks. The emergent interplay between standards development and regulatory approaches will be a decisive factor, and multiple, clashing standards and regulations might stifle progress in the area. Regulatory enforcement of standards may be criticized for stifling innovation, whereas too laissez-faire an approach may yield unintended consequences, as discussed in this section. Non-governmental organizations such as the Algorithmic Justice League have already helped scrutinize and remove bias from a number of facial recognition algorithms used by the police force in the United States. Similar approaches can be taken in manufacturing. Researchers also suggested that independent audit firms could develop reviewing strategies for AI projects and make recommendations to their client companies about what improvements to make. The insurance industry could also help guarantee trustworthiness by specifying requirements for underwriting AI systems in manufacturing.

We therefore call for more research in understanding how manufacturing organizations exploit AI technology in ways that can breach human privacy and well-being, and what mitigation mechanisms and guidance can be designed to prevent such breaches:

• • RQ 17: Should we build a manufacturing sector-specific code of conduct that interprets and adapts existing legal instruments pertaining to the use of AI?

• • RQ 18: How will we ensure interoperability between the various AI regulatory frameworks and standards that are currently being developed in different regions?

• • RQ 19: How will multinational manufacturing organizations adopt differing standards in their supply chains?

Brundage et al. (Reference Brundage, Avin, Wang, Belfield, Krueger, Hadfield, Khlaaf, Yang, Toner, Fong, Maharaj, Koh, Hooker, Leung, Trask, Bluemke, Lebensold, O’Keefe and Koren2020) identified institutional governance to be another key candidate for improving ethical practices. They suggest visible leadership commitment, including regular review board meetings, annual, publicly available responsible AI reports, and reward mechanisms for responsible AI practices, can be valuable in increasing incentives in organizations. Brundage et al. (Reference Brundage, Avin, Wang, Belfield, Krueger, Hadfield, Khlaaf, Yang, Toner, Fong, Maharaj, Koh, Hooker, Leung, Trask, Bluemke, Lebensold, O’Keefe and Koren2020) also suggest inter-institutional reporting mechanisms such as NASA’s Aviation Safety Reporting System and the Food and Drug Administration’s Adverse Event Reporting System, and Bugzilla as useful models for technical reporting.

As seen above, a large number of privacy and ethical challenges can arise during the model training phase. Data collection and model training are intertwined when it comes to trustworthiness. While in Section 4.1.1 we highlighted that data needs to be bias-free and obtained under consensus, and be able to be scrutinized by the owners and generators of data who might be unskilled in AI, in this section, we raise additional questions pertaining to personal data that is used to train ML models. These include:

• • RQ 20: What is the definition of personal data in a manufacturing context? How can workers know how their data is used, and can they have a right to consent or decline the way their personal data is used?

• • RQ 21: Should policymakers build mechanisms to ensure that data that was originally collected for its purpose remains its purpose in a manufacturing context?

• • RQ 22: What robust institutional mechanisms can be put in place to empower employees to scrutinize AI models in an organization? What methods can be used by an unskilled workforce to check for algorithmic bias and fairness, as well as decision processes that impact them, which rely on AI?

• • RQ 23: Counterarguments on surveillance activities on suppliers and workers highlight that surveillance has always been practiced, and the only difference AI brings is scale and accuracy. Does the manufacturing community, including generators of personal data, agree with this statement? In a manufacturing context, can we achieve consensus on what types of data collection and algorithmic surveillance constitute fair and unfair outcomes?

• • RQ 24: For unethical practices, how can the right balance between regulation and innovation be found? Should specific AI standards for manufacturing be built, and if so, should they be enforceable in different contexts?

4.4. Verification

Verification of AI models should include rigorous checks to ensure they are robust and reliable in satisfying functional and performance requirements. Here, robustness refers to the degree to which the developed model can function correctly in the presence of invalid inputs or varying environmental conditions, while reliability refers to the probability that the model performs required functions for the desired period of time without failure. Verified artificial intelligence has been defined as “AI-based systems that have strong, ideally provable, assurances of correctness with respect to mathematically-specified requirements” (Seshia et al., Reference Seshia, Sadigh and Sastry2022).

In the case of knowledge-based models, verification is rooted in the strong mathematical logic-based foundations of such models and leverages extensive research and development efforts in model checking (Clarke et al., Reference Clarke, Grumberg, Kroening, Peled and Veith2018) and automated theorem proving (Bibel, Reference Bibel2013), as well as formal specification languages (Baryannis and Plexousakis, Reference Baryannis, Plexousakis, Salinesi, Norrie and Pastor2013, Reference Baryannis, Plexousakis and Lomuscio2014; Baryannis et al., Reference Baryannis, Kritikos and Plexousakis2017). For instance, ontology-based knowledge models can be verified through an array of established reasoning systems, such as HermiT (Glimm et al., Reference Glimm, Horrocks, Motik, Stoilos and Wang2014) or Pellet (Sirin et al., Reference Sirin, Parsia, Grau, Kalyanpur and Katz2007), available through established, user-friendly tools, such as Protégé.

In contrast, verification of neural network-based ML models is a hard problem due to their black-box nature, making approaches such as the aforementioned model checking or theorem proving, or even source code reviews, not applicable (Salay et al., Reference Salay, Queiroz and Czarnecki2017). One of the issues is the large size of the state-space, making the design of test cases difficult. Reinforcement learning (RL) approaches especially suffer from large state spaces, as the decision space is non-deterministic and the system might be continuously learning, meaning that over time, there may be several output signals for each input signal.

Automated test case generation (Clark et al., Reference Clark, Kearns, Overholt, Gross, Barthelemy and Reed2014), transfer learning and synthetic data generation (Borg et al., Reference Borg, Englund, Wnuk, Durán, Levandowski, Gao, Tan, Kaijser, Lönn and Törnqvist2018) have been proposed as potential solutions. El Mhamdi et al. (Reference El Mhamdi, Guerraoui and Rouault2017) suggested that the robustness of a deep neural network could be evaluated by focusing on individual neurons as units of failure. Continuous monitoring of model input (elaborated further in Section 4.5), and integrating verification processes across the whole development cycle rather than at the end (e.g., by experimenting how output varies in the state-space with different model architectures) have been proposed as best practice (Taylor, Reference Taylor2006). Adler et al. (Reference Adler, Feth and Schneider2016) proposed coding a “safety-cage,” where the model execution is turned off with increased uncertainty and swapped with a deterministic model track. Although these suggestions stem from the field of autonomous systems, notably autonomous vehicles, they are worth noting and re-interpretation within the context of manufacturing is worth exploring.

Test-based verification usually involves the setting up of a simulation-based test environment, which is typically safer, cheaper, and faster to run. However, as with any simulation-based methodology, conclusions derived are dependent and constrained by the assumptions made by the simulation designer. Even small discrepancies between the simulation environment and the real world can cause dramatically different outcomes, exemplified by high-profile cases in the field of autonomous vehicles (Dulac-Arnold et al., Reference Dulac-Arnold, Levine, Mankowitz, Li, Paduraru, Gowal and Hester2021). Focusing on RL, the authors highlight several challenges with the transfer of RL algorithms from simulation-based training to real-life environments. These include limited sample size, delays in task rewards, constraints, unexpected, stochastic changes in the environment, and multiple objective functions. Stochasticity means that agents are not guaranteed not to explore unsafe conditions, which may have unintended consequences, unless these are thought by the designer in advance and coded into the reward function, which is often infeasible for the designer to capture exactly what they want an agent to do. Consider a robot tasked to pick up items in a warehouse from delivery zones and place them in designated locations. The algorithm designer may simply code a reward function to maximize the number of items picked and placed. In the simulation, the robotic agent works in a fairly constrained, stable environment. In reality, the layout of the warehouse may change, with moving obstacles that may include human workers. If the agent has not been trained to avoid moving obstacles, and its reward is based on number of tasks completed, it could explore taking unsafe shortcuts (called reward-hacking). While RL shows much promise as a learning paradigm, its implementations in real-world settings remain very limited (Z. Wang and Hong, Reference Wang and Hong2020). As RL is starting to be popularized in manufacturing robotics (Oliff et al., Reference Oliff, Liu, Kumar, Williams and Ryan2020), condition-based maintenance (Yousefi et al., Reference Yousefi, Tsianikas and Coit2020), vehicle-routing (Mak et al., Reference Mak, Xu, Pearce, Ostroumov and Brintrup2021), and inventory control (Kosasih and Brintrup, Reference Kosasih and Brintrup2022; Z. Wang and Hong, Reference Wang and Hong2020), it is worth exploring these challenges in real-world manufacturing environments.

ML performance metrics should be carefully considered in the application context. While the accuracy of a classifier is a well-known and widely used metric, in safety-critical applications such as machine failure, other metrics may be more appropriate (Baryannis et al., Reference Baryannis, Dani and Antoniou2019). For instance, recall may be more important, which measures the number of correctly identified positive classes over all classes that should have been identified as positive. In other cases, where incorrect identification of a class is costly, precision may be used—for example, when a production or quality delay is falsely predicted, resulting in inventory build-up. Other considerations may include quantifying the uncertainty of predictions, both in a classification and a regression context. Hence, performance metrics should be carefully designed and reflect contextual priorities. Additionally, performance checks should include checking for bias and fairness to overcome some of the issues relating to unfair outcomes, discussed in previous sections. Here inherent bias in the training data can be checked by comparing the ranges of features to the actual distribution of the feature in the real world across different data slices.

The main concerns at this phase include the safety and reliability of the developed models in noisy manufacturing contexts:

• • RQ 25: How can established approaches in knowledge-based AI verification, such as model checking, be leveraged in the case of ML?

• • RQ 26: How can test case generation methods developed be applied to manufacturing use cases?

• • RQ 27: How can the stochasticity of manufacturing environments be captured in ML test environments in a meaningful way?

Further, during the training phase, performance metrics can have varying impact on safety, cost, and efficiency. We ask:

• • RQ 28: How do different performance metrics affect outcomes in differing manufacturing contexts?

• • RQ 29: How can insurance and/or legal coverage be ensured for continuously adopting AI models?

4.5. Model deployment

Model deployment refers to the operationalization of the model that has been built, by building the software infrastructure that is necessary to run it, and setting and following policies on model maintenance and updates. One of the major challenges in this stage is identifying when a model needs to be updated with new information, and to what extent older information should continue to be utilized. In the case of ML models, “concept drift” describes the situation where the feature distribution shifts over time due to a change in the underlying data-generating process. Concept drift means that the mapping between features and output no longer matches the new incoming data. Thus, as real-world contexts evolve and adapt to changes over time, the underlying datasets that are representative of the system should change. For example, a demand forecasting model used to predict demand for a fashion product should be retrained frequently. However, in other cases, the changes in the system may not be contextually obvious to the model owner, in which case input data should be continuously monitored.

The two main ways to deal with concept drift are to update the model incrementally or to retrain the whole model, which can be done periodically, based on predefined performance criteria (such as accuracy or F1 scores) or statistical approaches based on uncertainty quantification (such as Hoeffding bound). Concerns akin to concept drift also apply to symbolic systems, where rules may degrade in effectiveness as operational contexts shift, necessitating mechanisms for verification, updating, or human-in-the-loop validation.

Through this illustrative example, it is made clear that the model deployment phase is continuous. Challenges resulting from this involve finding the right frequency and method of model updating and detecting when a model is no longer applicable to the context it is deployed in:

• • RQ 30: Which applications in manufacturing are prone to concept drift? Which drift detection methods are more informative in manufacturing?

• • RQ 31: What are the best practice mechanisms to identify models used in manufacturing that are no longer useful and should be updated or decommissioned?

5.1. Affordability

A key issue in adopting AI technologies is cost, which not only includes time and effort spent across the development and deployment steps of AI, but also the cost of data access, storage, and post-deployment costs such as maintenance. Studies performed on the adoption of digital manufacturing technologies, which include AI, show that adoption is often contingent upon affordability. In the UK, for example, over 99% of businesses are small to medium enterprises (SMEs, 0 to 249 employees) with lower affordability, which might create a larger capability discrepancy in supply chains. It is worth noting that many SMEs are vital to the supply chains of larger organizations, hence the success of the manufacturing industry is intertwined. The manufacturing community needs to monitor and encourage SME adoption, and we propose the following research questions in this context:

• • RQ 32: How can SMEs access the benefits of AI solutions?

• • RQ 33: Does the affordability of AI impact trustworthy AI adversely?

5.2. Outsourcing AI as a service

While no current statistics exist on the extent to which AI is developed in-house versus bought as-a-service, both approaches are not without challenges for manufacturers, and outsourcing decisions are likely to depend on a number of factors including the specificity of development, longevity of its use, AI skills the company would like to retain, the degree of control a company wants to exert on the algorithmic approaches developed and external infrastructure dependencies. While classical theories such as Resource-Based View and Transaction Cost Economics may offer useful starting points for framing AI outsourcing decisions, additional considerations on the trustworthiness of algorithms may need to be factored in. As mentioned earlier, outsourcing AI may make it more difficult to investigate bias in data used to train algorithms, as well as affect algorithmic safety. Moreover, it might be tempting to outsource in an attempt to try and shift accountability and responsibilities elsewhere, which raises legal and broader governance considerations (Cobbe and Singh, Reference Cobbe and Singh2021; Cobbe et al., Reference Cobbe, Veale and Singh2023) We call for more research on AI outsourcing decisions:

• • RQ 34: Are existing decision frameworks for outsourcing applicable for AI as a service in manufacturing and if not, how should they be extended?

• • RQ 35: How can trustworthy AI be ensured when outsourcing AI in manufacturing?

5.3. Data compensation and monetization

Data is one of the most valuable assets of firms that fuel much of the digital economy today. Often, datasets are traded amongst companies by so-called data brokers. As a society, we often do not know how and where our data is used, and for what purpose, forming an active field of ethical and regulatory debate. In manufacturing scenarios, companies may use data not only from individuals (such as monitoring how a consumer uses their products, or workers producing them), but also from other companies (such as monitoring activities of their suppliers and competitors). At the moment, generators of these datasets are typically not compensated. While this constitutes an ethical issue, in other cases data compensation may open up new markets and opportunities. For example, manufacturers may be interested in tapping into other companies’ datasets that they lack. For example, if a manufacturer would like to implement prognostics for its machinery but does not have enough run-to-failure data, it may be able to appropriate datasets from another manufacturer using the same machine. The data owner may wish to monetize its datasets. The automotive industry has been discussing how customers can be compensated by sharing car usage data so they can design better (McKinsey and Company, 2020). Data compensation is a strongly debated field where no regulatory guidance currently exists. Researchers have proposed the use of blockchain for tracking how data is used and developing mechanisms to compensate owners of data (Maher et al., Reference Maher, Khan and Prikshat2023). We call for more research on how manufacturers can monetize their datasets and also compensate data owners ethically and responsibly:

• • RQ 36: Which types of manufacturing data can be monetized?

• • RQ 37: Would monetizing data have an effect on AI trustworthiness in manufacturing and what are the associated risks and legal implications?

5.4. Scalability of trustworthy AI

Although some large manufacturers and supply chain businesses have made some advances in the incorporation of AI-based solutions into their decision-making and process control, for most companies (particularly SMEs), AI systems remain at the pilot stage (a situation often referred to as “pilot purgatory”). These pilot studies, as is the case with most AI solutions, implement models tailored to a specific use case and developed with data that manufacturers are typically not keen to share. However, as seen in the success of AI in other sectors such as finance or e-commerce, the secret to scalability and production-ready models is in sharing these models and data across businesses. Hence, to achieve the full potential of AI in the manufacturing value chain, it is recognized that models and data need to be transparent and usable, but at the same time secure to protect intellectual property and privacy (Davis et al., Reference Davis, Biller, St Pierre and Jahanmir2022). More research and development is needed to better understand the positive and negative effects of provenance on privacy and finding the best ways to develop and share models from aggregated data without losing sight of security issues:

• • RQ 38: How can trustworthy AI solutions be made more scalable in manufacturing?