1. Introduction
Need analysis is a foundational process in designing and developing effective systems, products, and services. Rooted in design science, need analysis involves identifying explicit and latent user needs to inform structured design decisions, much like requirement engineering in software or product development (Reference Baskerville and Pries-HejeBaskerville & Pries-Heje, 2010). In modern engineering practice, especially in system design and knowledge-based applications, understanding user needs is critical for creating scalable, efficient, and adaptable solutions. Needs analysis links solutions to real user experiences to make the design outcome innovative and practical.
In engineering contexts, need analysis serves as an essential step in the iterative refinement of knowledge management (KM) systems (Reference Basak and BhaumikBasak & Bhaumik, 2024). The increasing reliance on artificial intelligence (AI) and large-scale automation in engineering disciplines has led to the adoption of AI-assisted methods for requirement gathering, decision-making, and system optimization. Applying Large-Language-Models (LLMs) to automate need analysis mirrors the use of generative algorithms in design engineering, where computational models refine solutions based on evolving constraints and requirements (Reference ThimblebyThimbleby, 2008). In fast-paced and highly competitive industries, understanding user needs provides a source of competitive advantage, thus enabling organisations to differentiate their offerings and position themselves at the bleeding edge of emerging trends.
The paper will first look at the need analysis in KM and its implementation challenges, raising the gaps and the questions to be solved in this research. Then, the literature review section presents need analysis approaches and existing analysis automation techniques using LLMs, as the foundation of the research methodology. Finally, this paper introduces the prototype design and the results from tests and experiments to discuss the responses to research questions.
1.1. Need analysis in knowledge management
Generally, KM is identifying, organising, storing, and disseminating information within an organisation (IBM, 2024). The goal of KM studies is always to figure out the best practice for creating, capturing, organising, and sharing knowledge, thereby enhancing decision-making, innovation, and competitive advantage, creating value for human intelligence (Reference Dhar, Vaidhyanathan and VarmaDhar, Vaidhyanathan & Varma, 2024). KM is crucial for organisations to acquire, reflect, and develop their core competencies to maintain competitive advantages in the knowledge economy (Reference Jarrahi, Askay, Eshraghi and SmithJarrahi et al., 2023).
Without a proper need analysis, the design of KM processes may become less effective, reducing the efficiency of creating intellectual capital and thus restricting the organisation’s performance (Abualoush, Bataineh & Alrowwad, 2018). Figure 1 shows the necessity of need analysis in the overall KM methodology to guarantee the most appropriate and suitable design of the later phases, such as modelling and evaluation (Reference Mirafzal and Stal-LeMirafzal & Stal-Le Cardinal, 2024). A practical need analysis ensures that KM efforts are aligned with real-world requirements and priorities. It allows the organisation to identify gaps in knowledge, assess the requirements for system improvements, and ensure that KM systems are aligned with the actual needs of users (Reference DuDu Plessis, 2007).
Figure 1. Need analysis in KM methodology
Customisation based on the organisation’s specific context is indispensable when designing KM practices. Need analysis can serve as a critical way of guiding the customisation approach. Moreover, in developing and implementing a knowledge portal for higher education institutions, Aulawi et al. (Reference Aulawi, Ramdhani, Slamet, Ainisyifa and Darmalaksana2017) emphasised the importance of need analysis to identify the specific functional requirements and ensure that the knowledge portal addresses both individual and organisational needs. Thus, need analysis can give clues about the features that should be designed in the KM approach to satisfy the stakeholders’ expectations and, at the same time, indicate the potential concerns of the users, which should be addressed intentionally to avoid future resistance in implementation.
1.2. Challenges of KM in consulting firms
In the KM context, consulting firms are pivotal as knowledge-intensive organisations. For consulting firms, where knowledge is highly related to competitive advantages, the knowledge held within their organisations is an intangible asset. This knowledge can be in the form of competencies necessary to complete a task or experience gained over a long period. The benefits of effective KM practices are multi-faceted as they enable leveraging collective expertise to deliver tailored solutions (Reference DunfordDunford, 2000). For example, suppose the knowledge within the organisation can be consolidated, and the knowledge the consultants possess can be stored. In that case, even if there is a staff turnover in the future, the company will retain the expertise and experience (Reference Rehman, Bresciani, Ashfaq and AlamRehman et al., 2022). In addition, based on the wealth of augmented knowledge, a reliable and trustworthy image can be established to clients and demonstrate professionalism (Reference Kordab, Raudeliūnienė and Meidutė-KavaliauskienėKordab, Raudeliūnienė & Meidutė-Kavaliauskienė, 2020).
However, consulting work’s dynamic and project-based nature introduces unique challenges to KM (Reference Bolisani, Kassaneh, Lis, Scarso, Stolarek-Muszyńska and ZiębaBolisani et al., 2023). These firms often face high employee turnover, diverse client demands, and the need to quickly adapt to evolving market conditions, making it challenging to maintain cohesive and accessible knowledge repositories. Besides, although new practices and methodologies for KM are continuously put forward, in the intelligent era with a more complex and ever-changing knowledge environment, most companies face the dilemma of obtaining low returns with high investment in KM and need help to transform KM practice into satisfying performance successfully (Reference Vadari and DesikVadari & Desik, 2021). Additionally, KM professionals with no expert technical skills find it even more challenging to interpret the data collected. Those who might be performing the KM processes at an operational level must gain the competence to analyse big data or dissect complex survey findings. The reliance on manual methods increases the likelihood of human errors and may decelerate the prompt extraction of actionable insights. Concerning these matters, academics have increasingly adopted automation with advanced computational technologies.
In this background, Jarrahi et al. (Reference Jarrahi, Askay, Eshraghi and Smith2023) highlighted the shift from rule-based to data-driven KM when dealing with new knowledge, which reduces human participation but increases the volume of data to be analysed. Anshari et al. (Reference Anshari, Syafrudin, Tan, Fitriyani and Alas2023) also concluded from a comprehensive literature review that KM is under transformation as complex knowledge tasks are gradually automated, and the adaptability of the KM system is increased. This trend motivated researchers to look for newer technologies to make KM processes more adaptable and flexible (Reference HeikkinenHeikkinen, 2024).
1.3. Research gaps and research questions
In consultancy firms, the rush to complete projects quickly sometimes overlooks the need for analysis. Pressed for time with few resources, teams may skip any deep needs analysis. On the one hand, effective KM methods rely on accurate need analysis to identify gaps in knowledge creation, sharing, and application. However, traditional need analysis methods in KM often depend on time-consuming manual processes, qualitative assessments, and expert interpretation, making them prone to subjectivity and inefficiency. The increasing complexity of knowledge environments, particularly in consulting firms, further exacerbates these challenges.
On the other hand, AI has been increasingly applied in automating decision-making and pattern recognition in data-driven fields, yet its application in KM remains underexplored. LLMs, integrated within Retrieval-Augmented Generation (RAG) frameworks, offer a novel approach to automating need analysis by efficiently extracting and synthesising knowledge from large textual datasets (Reference Freire, Wang, Foosherian, Wellsandt, Ruiz-Arenas and NiforatosFreire. 2024). Therefore, this research hypothesises that an LLM-based agent can significantly improve the accuracy and efficiency of need analysis in KM compared to traditional manual methods. Specifically, we propose that integrating LLMs within an RAG framework will enhance need identification, reduce human effort, and provide insights aligned with expert evaluations. To test this hypothesis, we will assess the model’s performance on real-world datasets, comparing its results with those obtained through conventional need analysis approaches.
Based on the previous discussion, the study comes up with the following research questions:
-
RQ1: How can LLMs effectively be integrated into KM practices to automate need analysis?
-
RQ2: What is the performance of an LLM-based agent compared to traditional manual Need Analysis methods?
2. Need analysis literature review
Needs analysis is a formal process of identifying, measuring, and prioritising the targets’ needs. It begins with information collection, which is a critical step in data analysis. Raw information is collected to give a complete understanding of the situation and the problems the users face (Reference DunfordDunford, 2000). Ideally, for effective information gathering, planning must be done carefully so that the information captured will be relevant, representative, and adequate to meet the analytical requirements (Reference Nagarajan, Ganesh, Punniyamoorthy and ResmiNagarajan et al., 2012). The analysis follows after information gathering, where valuable insights are sought from the data collected. Data analysis includes identifying patterns, relationships, and trends that may point toward key user needs and pain points (Reference GrantGrant, 2002). The objective is transforming raw data into useful information, which implies that results must be clear-cut and directly related to the needs of intended users.
2.1. Need analysis approaches
KM engineers employ various approaches to conduct need analysis during the design and implementation of KM systems. These methodologies have been developed to discover explicit and tacit knowledge sources, understand existing practices, and align KM systems with organisational strategy. Conventional needs analysis methodologies include questionnaires and interviews, which provide direct ways of collecting data from the stakeholders (Fowler, 2013).
According to Mirafzal, Wadhera and Stal-Le Cardinal (Reference Mirafzal and Stal-Le2022), interviewing is currently the most used technique for need analysis. Interviews provide direct interaction between KM engineers and consultants, managers, or other stakeholders who can offer valuable insights into what they did, do, and want to achieve. The selection of the right interviewees is important, considering that they have different perceptions that build on the effectiveness of the KM system. Observation is another common approach, especially within consultancy firms, by being plunged into the work environment to see how knowledge is created, shared, and used. Questionnaires are used together with interviews and observations to solicit systematic feedback from large pools of participants to enable the engineers to identify patterns and trends in KM needs.
Meanwhile, documentation review is vital for KM engineers to review existing documents, reports, and project files and establish best practices for substantial insight. It helps to identify shortcomings in the present documentation and ways to improve it. Also, innovative technologies are fast enhancing the more traditional approaches. For example, Natural Language Processing (NLP) can analyse unstructured textual information from interviews and documents, synthesising themes quickly and highlighting areas where further consideration is called for (Reference Mirafzal, Wadhera and Stal-LeMirafzal, Wadhera and Stal-Le Cardinal, 2022).
Moreover, new technology has increased the effectiveness of questionnaires and personal interviews Karunarathna et al. (Reference Karunarathna, Gunasena, Hapuarachchi and Gunathilake2024). The rapid development of technology, accompanied by the emergence and advancement of Gen AI, presents new opportunities for transforming KM practices in organisations. With abundant data and constantly enhancing computing power, AI tools are increasingly penetrating the internal organisation of companies (Reference Dhar, Vaidhyanathan and VarmaDhar, Vaidhyanathan & Varma, 2024). Under the operation of Gen AI, intelligent systems can be designed to perform autonomous tasks, such as generating knowledge and answering problems (Reference Bhupathi, Prabu and GohBhupathi, Prabu & Goh, 2023), which may help need analysis. Although these enhanced tools are still not widely applied within KM systems, their application in related domains showed that they undoubtedly could revolutionise the need for analysis by reducing manual effort and increasing analytical precision. Thus, using LLM to achieve high-performance and efficient output can be an essential topic in needs analysis.
2.2. Analysis automation using LLMs
Some researchers have already made some experiments on analysis automation using LLM, which can offer insights into how we can automate a need analysis approach. For example, Hutchinson et al. (2024) revealed that LLMs were used to automate data preprocessing tasks, such as schema matching, data imputation, and error detection, all critical data management elements. Automating these processes has enabled data analysts to focus more on complex analytical tasks rather than tedious preparatory tasks, thus increasing efficiency and productivity and fitting the needs in the KM context. Also, the Chat2Data initiative by Zhao et al. (Reference Zhao, Zhou and Li2024) highlighted how LLMs can serve automated data retrieval and processing by translating a natural language input into SQL queries or commands for visualisation, showing the possibility to conduct complex data analysis without requiring users to have technical knowledge.
Similarly, Rasheed et al. (Reference Rasheed, Waseem, Ahmad, Kemell, Xiaofeng, Duc and Abrahamsson2024) present a vision of multi-agent systems based on LLMs that automate qualitative data analysis, which has traditionally been very labour-intensive. This study showed how LLMs could be used to automate content analysis, code generation, and the interpretation of qualitative data, ultimately streamlining the data analysis process in fields like software engineering (SE). Using a multi-agent system makes the approach scale up and increase efficiency, significantly reducing the time and effort needed to analyse data. This becomes particularly important in the domain of KM, where the ability to efficiently analyse large, unstructured datasets, like interview transcripts or survey responses, becomes critical.
RAG is one of the most critical elements for LLM-based automation systems of KM practices, requiring efficient information retrieval and contextual awareness among their required responses. Zhong et al. (Reference Zhong, Xin, Li, Zhan and Cheng2024) proved that integrating RAG increases collaborative problem-solving by providing higher contextual accuracy and reliability in the responses. Their study indicated that using RAG with LLMs will significantly raise the ability of agents to provide relevant support, therefore favouring learning results, which could be implemented in a need analysis setting.
3. Research methodology
This research adopts a mixed-methods approach, combining qualitative and quantitative studies to assess the system comprehensively. The research framework will enable the researcher to adopt various methodologies and techniques for data collection, with the assurance that the research will maintain the necessary flexibility to cover the complexity surrounding implementing LLM-based systems in organisational settings.
3.1. Data collection
The design of the data collection process is very careful to expose the LLM-based system to a variety of inputs, and to distil potential design improvements of the model based on the system logs. This research collected and used the following three types of data:
-
The real-world datasets, which include Excel files, collected questionnaire responses from stakeholder interviews.
-
Synthetic data generation by ChatGPT: this involves the input of various scenarios related to need analysis to generate textual data mimicking stakeholder feedback in a KM environment. The purpose of collecting synthetic data is for the initial model test while adjusting the functions and the flow.
-
System logs created by the LLM-based system when performing tasks of need analysis.
The interviews were conducted at a consulting company. A total of 10 managerial stakeholders were interviewed for this study, representing a range of roles within the consulting firm, from the Chairman, CEO, and General Secretary to various Vice Presidents overseeing functions such as Research and Innovation, Industry, Services, Transformation, and Finance. The targeted participants in this study provide variable dimensions of perception concerning KM, anchored in their different responsibilities and areas of concern inside the organisation. For interviews with consultants, this study chose to use the first-hand questionnaire responses collected by Mirafzal and Stal-Le Cardinal (Reference Mirafzal, Wadhera and Stal-Le2024). There were 20 responses collected in total in the form of Excel. Answers were anonymised so that the privacy and confidentiality of the respondents were ensured, and personal data was not included in the analysis. The data collected were stored securely, with access restricted to the research team to prevent any unauthorised disclosure of information. The diversity in data types allows a more comprehensive testing of the functionalities of the system.
3.2. Data analysis methods
The data analysis involves both qualitative and quantitative methods. Qualitative analysis focuses on evaluating the relevance and accuracy of the insights generated by the LLM-based system. This is done manually by analysing the reports generated by the system compared with those previously generated by human experts. The qualitative analysis also includes feedback from KM professionals regarding the utility and comprehensibility of the reports produced by the system.
Quantitative analysis measures the system’s performance in terms of speed and accuracy. Metrics such as the time taken to complete need analysis tasks, the number of correctly identified stakeholder needs, and the overall precision of the insights are used to evaluate the LLM-based system. Meanwhile, ethical considerations are paramount during the analysis. Informed consent is obtained from all participants involved in the real-world dataset, ensuring that they know how their data will be used and that they retain the right to withdraw from the study at any time. The synthetic data generated by ChatGPT does not raise ethical concerns since it is not associated with real individuals or organisations. Data security and confidentiality are maintained throughout the study, with all data stored securely and anonymized before analysis.
4. Prototype design
In this section, we propose a proof of concept to illustrate the use of the methodology presented in Section 3.
4.1. Design philosophy
The ultimate purpose of this prototype is to automate the need analysis processes with high accuracy. To do so, we integrate both approaches of NLP analysis automation and RAG structure.
On the one hand, we set up two agents, one named Intent Recognition Agent, calling for related NLP analysis functions and another named Consultant Agent to generate insights from the given repository. On the other hand, we utilise the RAG structure to let the model answer user follow-up queries.
The prototype can rely on both the results from the Consultant Agent and Intent Recognition Agent to provide more comprehensive insights. It will also be interesting to observe whether and how the influence generated by the NLP analysis will impact the final conclusion of consultant needs provided by the model. This hybrid approach integrates traditional text analysis techniques, such as thematic and sentiment analysis, with modern LLM capabilities to produce actionable insights. The process should ensure scalability, adaptability, and precision in conducting need analyses based on diverse textual inputs.
4.2. System architecture proposition overview
The overall structure depicted in the flowchart (See Figure 2) outlines the integration of an LLM-based RAG framework with dedicated agents to streamline the analysis of responses.
Figure 2. Flowchart of the model
To address inefficiencies in KM need analysis, this study designs an LLM-based system structured into two layers, each serving a distinct analytical function (in Figure 2). The first layer, the Consultant Agent, processes structured questionnaire data, determining whether consultant-specific needs should be prioritized. This is different from directly letting the model read the original responses and give answers. Instead, it enhanced the reliability and quality of the insights generated by the model with scientific evidence provided by the NLP results.
The second layer, the Intent Recognition Agent, applies NLP-based classification techniques to extract relevant insights from textual inputs. It is a valuable experiment as one further step taken based on the previous research from Mirafzal and Stal-Le Cardinal (Reference Mirafzal, Wadhera and Stal-Le2024) in the same consulting company. Their study proposed the definition of six need types for consultants, and after manually analysing the questionnaires collected from consultants, they already demonstrated findings of the need analysis. This could be regarded as existing results for comparison, examining the performance of this need analysis model.
This architecture allows the system to refine need identification from interviews to definitions of insights, incorporating both structured and unstructured knowledge sources and ensuring context-aware analysis. These agents are a critical part of the system aimed at automating specific tasks, facilitating seamless interaction, and improving knowledge retrieval, application quality, and efficiency.
After receiving the results from the optional needs retrieval for consultants in the first layer, and the mandatory intent-driven NLP analysis from the second layer, the system leverages RAG to handle queries and follow-up questions dynamically, ensuring contextual relevance in retrieved data and subsequent analysis. Insights generated from this workflow, therefore, embrace both the context from the given database and the accurate NLP analysis results, which can be regarded as factual evidence.
5. Result
The main NLP analysis functions used were topic modelling using TF-IDF, sentiment analysis, keyword extraction, and word cloud to identify the managerial stakeholders’ perceptions regarding KM.
5.1. Topic modelling results
We first use synthetic datasets to test the performance of the prototype. For each dataset, we first invited one expert to finish the NLP analysis and drew insights from the results. Referring to the records in the system logs, several adjustments were made to the prompting of the agents based on the results. The content of the retrieved documents, which explained needs in KM, was also revised with more examples for the agents to understand fully. As satisfactory results were achieved, we replaced the synthetic datasets with real-world datasets collected from interviews. Using the topic modelling function, for each question category, 2 topics were identified and named based on human experience. When asking the need analysis model to do the same, the responses from the system were close to those from human work.
The below comparison (See Figure 3) shows the differences and links. Although human work and model work didn’t give the topics exactly the same names, we can still observe their internal connections. When referring to the short descriptions LLM gave for each topic, it is not hard to find that it prioritised very similar KM needs as the ones decided by humans.
Figure 3. Topic modelling results
Regarding the final input generated by the model, which delivers the overall insights from the need analysis, we find that the need gaps identified are highly close to those identified manually. It is also worth noting that the model could identify needs gaps humans identified as potential risks. Overall, the results generated by this need analysis model deliver the same or even above quality.
5.2. Need analysis results
To test whether the Consultant Agent can enhance the model’s performance when analysing the needs from questionnaires with the consultants, we use the first-hand questionnaire responses from the work of Mirafzal and Stal-Le Cardinal (Reference Mirafzal, Wadhera and Stal-Le2024). They prioritised the type of needs: Business Knowledge, Collaboration Needs, Technical Knowledge, Experience/Know-how, and Theoretical Knowledge.
After continuously adjusting and optimising the prompt and the documents in the RAG repository, we successfully got results that were pretty aligned with those given by human experts (Figure 4). The main difference lies in the need for reusable components and theoretical knowledge. However, since the importance of these two types of needs was insignificant in both approaches, it didn’t challenge the overall conclusion of the need types identified, which recognised this agent’s performance. With this list of need types identified, when processing into the Intent Recognition Agent and continuing the follow-up analysis, the model offered more comprehensive and reasonable insights and suggestions for addressing these need gaps for consultants.
Figure 4. Results generated by the Consultant Agent
6. Discussion and conclusion
This research has demonstrated the potential of using LLMs in KM, both in NLP analysis and RAG architectures. The proposed system is based on a two-layered architecture involving specialised agents regarding various aspects of the needs analysis process. Regarding RQ1, the findings confirm that LLMs can be integrated into KM practices using a structured, agent-based approach combined with a RAG framework. The model automates need extraction through NLP techniques, improving consistency and scalability compared to manual analysis. The Consultant Agent and Intent Recognition Agent enhance contextual awareness, allowing for more targeted and relevant insights.
For RQ2, the LLM-based agent demonstrated high alignment with human assessments in need classification and prioritisation. The model effectively identified stakeholder needs, with results comparable to expert analysis. While minor discrepancies were noted in prioritising theoretical knowledge, the automation process significantly improved efficiency, reducing the time required for need analysis. These findings highlight the potential of LLM-based systems in streamlining KM processes while maintaining analytical accuracy.
In conclusion, the findings confirm that LLMs can be effectively integrated using a RAG-enhanced framework, where dedicated agents process, classify, and retrieve relevant need-related insights from structured and unstructured data sources. This approach ensures alignment with organizational contexts while reducing the cognitive load on KM practitioners. Meanwhile, the results validate that when properly structured within an engineered framework, LLMs can support and augment KM processes, particularly in consultant-driven environments where rapid need assessment is crucial.
Furthermore, there are limitations concerning data input. Currently, the model is structured to process data predominantly in Excel format. While this format is appropriate for answers based on structured questionnaires, more is needed for organisations relying on other data formats. The quality of the raw data recorded from the interviews in Excel is also vital.
Finally, this research can form a basis for the extension of the capabilities of the system in the domain of KM beyond the need for analysis to other KM activities such as knowledge retrieval, sharing, and generation. Addressing the identified limitations of this research and expanding its applications will further contribute to developing innovative AI-driven solutions that will enhance decision-making, innovation, and competitive advantage in KM practices.
Open access
