1. Introduction
Over the past quarter century, the mystery behind the innovation process has been informed and formalized through a deeper understanding of the cognitive processes for design (Reference Tseng, Moss, Cagan and KotovskyTseng, et al., 2008; Reference Linsey, Tseng, Fu, Cagan, Wood and SchunnLinsey, et al., 2010; Reference Sio, Kotovsky and CaganSio, et al., 2015; Reference Hey, Linsey, Agogino and WoodHey, et al., 2008; Reference Yu, Honda, Sharqawy and YangYu, et al., 2016), computational search (Reference Cagan and KotovskyCagan & Kotovsky, 1997; Reference Cagan, Campbell, Finger and TomiyamaCagan, et al., 2005; Reference McComb, Cagan and KotovskyMcComb, et al., 2015, Reference Lapp, Jablokow and McCombLapp, et al., 2019), process methodology (Reference Cagan and VogelCagan & Vogel, 2002; Reference Otto and WoodOtto & Wood, 2000; Reference Ulrich and EppingerUlrich & Eppinger, 1995; Reference AltshullerAltshuller, 1996; Reference Hauser and ClausingHauser & Clausing, 1988), and teaming (Reference Fu, Cagan and KotovskyFu, et al., 2010; Reference Gyory, Cagan and KotovskyGyory, et al., 2018; Reference Toh and MillerToh & Miller, 2015; Reference Fu, Cagan and KotovskyFu, et al., 2010). With regard to creating innovations, the needs, wants and desires of the user and other key stakeholders has been identified as a critical driver (Reference Cagan and VogelCagan & Vogel, 2002). In Creating Breakthrough Products, Reference Cagan and VogelCagan and Vogel (2002) Footnote 1 introduced the Value Opportunity Analysis (VOA) method as a way to channel the primary stakeholder insights acquired during product development research into an articulation of what the stakeholders value in a product (physical or service) solution. The VOA method emerged from research into the best practices of innovative companies at that time (late 1990s). The result of the VOA is a chart (Figure 1a) that indicates what value opportunity attributes are relevant and how important they are to the stakeholders. The chart, coupled with the knowledge gained from the interviews, surveys and secondary research from news, the literature, and data, directs the articulation of the requirements to meet the emerging value proposition of a product to meet those needs, wants and desires of the stakeholders, and succeed in the marketplace. As an example, Figure 1b shows a chart filled in for the opportunity to “design a service that helps mobility impaired people to get around.” Of note this chart is the result of a collaborative process between the user and the VOA LLM Bot described in the remainder of this paper, and presented as a case study later in the paper.
Value Opportunity Analysis (VOA) chart: (a) generic and (b) example for the case study in this paper, namely for the opportunity to “design a service that helps mobility impaired people to get around.”
The VOA method has been widely used over the past 25 years both as an approach to teach innovation to students, but also in industrial practice where companies have produced award winning products through its aid and have even re-organized their corporate structure based on the interdisciplinary framework in Creating Breakthrough Products. As well these ideas have permeated research and practice, even if not a direct application of the VOA method.
Although a powerful and meaningful method, the application of the VOA can be challenging, especially for novices in its use. The method takes effort and insight, converting qualitative data to succinct attributes and resulting requirements.
Over the past decade AI and machine learning have evolved based on processing at higher speeds, large and fast accessibility of data storage, and clever algorithms that unpack and filter data in new ways. In 2017, Vaswani et al. introduced the transformer architecture in their seminal paper “Attention is All You Need” (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and PolosukhinVaswani, et al., 2017). This architecture was subsequently built upon by researchers at OpenAI beginning in 2018, where they coined the term Generative Pre-trained Transformer (GPT) and developed their first large language model (LLM), GPT-1 (Reference Radford, Narasimhan, Salimans and SutskeverRadford, et al., 2018). These GPT models are what power OpenAI’s well-known LLM application ChatGPT and subsequent alternatives such as Anthropic Claude and Google Gemini. GPT’s work through three stages in the reverse order of the name: Transformer, Pre-training, and Generation. At the transformer stage, the neural network architecture is designed and implemented as a computational model with self-attention mechanisms ready to model language. Then in the pre-training stage, the model is trained on vast amounts of text to capturing language patterns over millions of iterations so that it can begin predicting language patterns. This is also the stage that the parameters of the neural network are derived and adjusted. Finally, in the generative stage, the fully formed GPT can generate coherent and contextually relevant text in response to user input. Most models today also go through reinforcement learning from human feedback, to fine-tune model behaviour and align it to developer preferences.
Widely accessible GPTs have transformed how people think about the potential and realities of machine learning. In a 2023 joint study conducted by Harvard Business School and Boston Consulting Group, Reference Dell’Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and LakhaniDell’Acqua, et al., (2023) studied the pros and cons of access to GPT-4 for 750 of their consultants. In those tasks where the researchers noted that GPT-4 has strengths such as ideation, writing, data analysis, knowledge workers saw a boot in productivity, 12.2% more tasks completed on average, 25.1% faster and a 40% overall increase in quality of work. However, for tasks that leverage AI outside its competencies such as contextual judgements and synthesis of diverse information there was a 19% drop in the ability to produce correct solutions. The team further reported that there was a 41% decrease in diversity of thought.
By accounting for these cons and leveraging these pros, we created and introduce a new tool to guide and inform the VOA process and provide data through a preliminary user study on effectiveness and impact on the product development process. The VOA LLM Bot leverages LLM technology to create intelligent agents that help guide users to create a VOA, perform a product opportunity gap context analysis including the options of comparing the goal of the effort to current products in the market and other current solutions, then create a product requirements document from which conceptualization can take place. The primary agent is trained on the literature, including appropriate sections of the book Creating Breakthrough Products, so that it has a deep theoretical and practical knowledge of the VOA process. The VOA LLM Bot guides designers through a scaffolded interview process and acts as a partner, helping ensure that each topic is covered through the process. The design partner is flexible and responsive to user input allowing for a dynamic interaction. Users have the autonomy to delve deeper into topics and are encouraged to experiment with applying the VOA process in whatever creative manner they feel helpful.
Beyond introducing a valuable design tool, this paper illustrates how LLM technology can augment the design process, providing boosts to performance and organization, allowing designers to potentially think more strategically. Although the VOA LLM Bot is useful in practice, the work also illustrates modalities in which LLMs can provide deeper capabilities and learning insights to students in their engineering and design education.
2. Value and product development
Value refers to the inherent meaning that people need, want and desire through experience, in this case through the experience from a product or service. The goal is for that valued experience to be useful, usable, and desirable. Through research on best practices from the most innovative companies, and their testing through their new product innovation classes, Reference Cagan and VogelCagan and Vogel (2002) introduced value opportunities (VOs), with specific attributes that contribute to a product’s usefulness, usability, and desirability and that connect a product’s features to that value. Products and services enable an experience for the user, so the better the experience, the greater the value of the product to the relevant stakeholders.
There are seven classes of Value Opportunities which encompass the value attributes necessary to consider in developing a product or service:
-
• The emotion that results from the perceptual experience of the consumer when using the product or service;
-
• The aesthetics that stimulate enjoyment or appreciation;
-
• The societal impact, through environmental and social factors;
-
• The brand identity of the product or service (and its possible relationship to the company);
-
• The ergonomics of use and interaction;
-
• The core technology or performance; and
-
• The quality of the production and offering of the product or service.
Ultimately value, and the meaning it represents to people, result in the emotions people feel with using the product or service. The ergonomics, core technology, and quality VOs each address the satisfaction of the product during use, while the social and environmental impact, product identity, and aesthetics VOs address lifestyle aspects of the stakeholder. Each of these VO categories can be broken down further as shown in the Value Opportunity Chart in Figure 1.
The VOs are all encompassing but generic. For application to a given product development task, these attributes need to be defined in the context of the product/service domain, enabling a tuned and focused value proposition. The VOs have been used to design and analyse hundreds of products in a variety of applications. More details can be found in Reference Cagan and VogelCagan and Vogel (2002).
The Value Opportunity Analysis (VOA) chart shown in Figure 1 helps the designer channel what they know about what their stakeholders value based on qualitative and quantitative user research, and secondary research from the literature. When filled in, the chart indicates which VOs are relevant to a product/service opportunity, and the level of their importanceFootnote 2 . An example of a filled in VOA Chart and resulting product specifications is seen in Figure 1b for the problem of designing a service that helps mobility impaired people to get around. The next step articulates why these attributes are relevant based on what they mean for the stakeholders and what a product or service must do to be able to meet the expectations of the stakeholder. That resulting list serves as a set of product design requirements to drive product ideation. The Case Study section presents one such set of product requirements generated with the VOA LLM Bot for this sample problem, as discussed in what follows.
Of note throughout the paper, products or/and services are referred to. At this level of opportunity specification there is not a significant difference between the two. Rather this method helps articulate what is required to solve a problem more generally. However, with the evolution of service design some terms are different between physical requirements and services or digital interaction, and these are presented in the VOA Chart as alternatives.
3. VOA LLM Bot
The VOA LLM Bot implements Value Opportunity Analysis through natural language conversation, using standard Large Language Model (LLM) technology. The system is built upon the Dietrich Analysis Research Education (DARE) Multi-Agent LLM framework designed at Carnegie Mellon University to support innovation and rapid development of web-based LLM applications (https://www.cmu.edu/dietrich/ai/education/index.html). The DARE system is a modular codebase that facilitates the orchestration of multi-agentic workflows or chatbots to create specific interactions. This work is implemented using the Anthropic Claude 3.5 model, built on a foundation of more than 175 billion text and code parameters
The VOA LLM Bot is a DARE implementation of an Agent+Helper framework (Figure 2). Its core functionality relies on two specialized intelligent agents working in tandem to support the VOA design process. The primary agent (Agent) serves as an AI design partner, leveraging custom crafted system prompts and retrieval augmented generation (RAG) to ground its responses in VOA methodology. RAG stores uploaded vectorized embeddings of the documents in a vector database, then performs a semantic similarity search against user input. It then retrieves relevant contextual chunks of text to augment the user’s input before sending it to the LLM, creating richer more relevant output and reducing the likelihood of hallucinations. The RAG implementation incorporates domain-specific knowledge from established VOA literature, enabling the agent to provide contextually relevant guidance.
Agent+Helper framework, illustrating how RAG knowledge base communicates with agent
A multi-step process, referred to as iterative agent alignment, is introduced. An in-depth assessment of the base model via the API to establish baseline capabilities, demonstrates the model to be very knowledgeable about the book Creating Breakthrough Products, likely due to its public availability. We therefore infer that it is likely that the book, or closely related material, is included in the model’s training data. After establishing this baseline, a system-prompted, prompt-only Agent is created to identify gaps in coverage, interaction design, and tone. Testing this agent determines deficiencies and iterates until performance can no longer be improved through prompting alone.
Next, selecting and creating relevant source materials to populate the Agent’s RAG database produces further improvement. The system prompt is then adjusted to create contextual scaffolding within the prompt to ensure improved retrieval and methodological coverage. This resulting system incorporates relevant excerpts from Creating Breakthrough Products into the Agent’s RAG store to provide grounding and reinforce alignment with the VOA method.
The secondary agent (Helper) functions as an assistant to the main Agent, in this use case serving as a progress monitor that analyses ongoing conversations to track advancement and completion through the VOA process. The Helper is set up using a system prompt that focuses solely on user input from the conversation with the main Agent to ensure accurate progress tracking. For more granular tracking, a creator can also define specific items, such as design goals, which can be added as supplemental context. It can track progress, act as a summarizer, display progress meters, or perform other functions as defined by the creator and expressed in markdown. In this use case, the Helper uses simple visual cues and descriptions to reduce the user’s cognitive load in tracking, so gaps can easily be identified to ensure completion, allowing them to focus on the main task.
Natural language processing forms the foundation for both agents’ functionality. The system processes the user’s input to connect their ideas to design-related concepts, maintains awareness of conversation context, and generates responses aligned with the VOA methodology. This system is highly adaptive to user input and deferential to the needs of the user. While the system is designed to efficiently guide a user through an interview within the VOA methodology, the method is flexible and users retain agency and autonomy over the design process, which we believe is critical to encouraging creativity. At all times, users have the choice to direct the bot to focus on different aspects of the method they prefer to focus on.
The user interface presents three distinct panels for interaction (Figure 3). The main conversation panel (centre) facilitates natural language exchange with the AI design partner (Agent). A dedicated progress tracking panel (left) displays metrics generated by the progress agent (Helper). The third panel (right) provides space for user notes and documentation.
VOA LLM Bot interface, indicating the helper and agent portions
Importantly, the LLM VOA Bot serves as tool to aid the user in defining and articulating the value proposition and resulting product requirements. Yet the Bot demonstrates power and insights in its ability to guide the user through the VOA process, leveraging not only the direct information provided by the user but also the broad base from the 175 billion text and code parameters of Claude, and delivering a robust set of product requirements when requested to do so by the user. Reference McComb, Boatwright and CaganMcComb, Boatwright and Cagan (2023) presented a matrix of different modalities for Human+AI collaboration, differentiating modality and focus of purpose. The LLM VOA Bot illustrates three different modalities of interaction: The Bot coaches users in the design process through real-time direction and guidance (proactive and process-focused); it independently contributes proactive and skilled suggestions toward designing (proactive and problem-focused); and it provides “just-in-time” information to users based on real-time value articulation progress (reactive and problem-focused).
4. Case study
To illustrate the use of the LLM VOA Bot, consider the opportunity to “design a service that helps mobility impaired people to get around”. The Bot asks for this high-level problem, and then starts working with the user to understand which value attributes from the VOA are relevant and why.
For example, from a conversation:
AI: Since you’re designing a service, let’s start exploring these VOA categories to better understand what matters most in your project. Let’s take them one by one.
For the Emotion category, which attributes (adventure, independence, security, confidence, power) do you think are most relevant to your mobility service for impaired people?
User: I think all except sensuality are relevant
AI: Thanks for sharing that! So adventure, independence, security, confidence, and power are relevant emotional attributes for your mobility service. Could you tell me a bit more about why these specific emotional attributes are important in the context of helping mobility impaired people get around? What do these emotions mean for your target users?
User: 1) everyone like to do fun things too, like seeing a show or going out 2) I think people want as much autonomy at all stages of their lives and it becomes especially acute as you may lose some capabilities 3) this is a vulnerable population and we want them to be able to trust that by using our service we are not putting them at risk 4) I remember getting my license and how it made me feel I could better navigate the world, I would imagine this would be similar 5) I don’t like having to rely too much on others for everyday tasks. This kind of service would give them back some confidence they are in control.
The result from that input is this response:
AI: Thank you for sharing those insights about your mobility service! Your points really highlight the emotional importance of mobility for people with impairments.
I can see how adventure is relevant - enabling people to enjoy fun activities like seeing shows or going out creates a sense of adventure and participation in life.
Independence is clearly central to your vision - helping people maintain autonomy even as they face physical challenges, reducing reliance on others for everyday tasks.
Security is crucial for this vulnerable population - building trust that your service will keep them safe, not put them at risk.
The confidence aspect really comes through in your license analogy - that feeling of being able to navigate the world independently creates a powerful sense of self-assurance.
And power - giving people control over their own mobility and daily choices rather than depending on others.
After working through all of the value attribute categories, the VOA LLM Bot assesses then summarizes the Value Opportunity categories as:
Emotion: HIGH
-
• Adventure: Medium - Enabling users to participate in enjoyable activities like shows or social outings
-
• Independence: High - Providing autonomy to mobility-impaired people, reducing reliance on others
-
• Security: High - Creating trust that vulnerable users will be safe while using the service
-
• Sensuality: Low - Not identified as relevant
-
• Confidence: High - Empowering users to navigate the world more independently
-
• Power: High - Giving users control over their mobility and daily choices
Aesthetics: MEDIUM
-
• Visual: Medium - Some visual identity is important but not critical (app interface, vehicle signage)
-
• Tactile: Medium - Clean and comfortable transport vehicles are a priority
-
• Auditory: Medium - Important but must account for hearing impairments
Identity: HIGH
-
• Personality: High - Service must be perceived as trustworthy and reliable
-
• Point in Time: Low - Not specifically mentioned as important
-
• Sense of Place: High - Building community connections through common routes to senior centers
Impact: MEDIUM
-
• Social: High - Addressing loneliness by facilitating community connections
-
• Environmental: Low - Not identified as an important factor
Interaction: HIGH
-
• Ease of Use: High - Multiple access methods (phone, app) with tracking capabilities
-
• Safety: High - Critical for helping users get in and out of vehicles safely
-
• Comfort: Medium - Standard vehicles are sufficient with proper accommodations
Performance: HIGH
-
• Enabling: High - Core purpose is empowering mobility-impaired people to get around
-
• Reliable: High - Absolute dependability is essential for vulnerable users
Quality: HIGH
-
• Consistency: High - Service must deliver the same quality experience every time
-
• Flexibility: High - Must adapt to different user needs and situations (like sudden weather events)
This results in the VOA Chart shown Figure 1b.
After providing information about three competitive existing services, the user requests a set of product design requirements, in this case only those associated with the “high” value attributes, resulting in the following output:
This set of requirements for a product (in this case service) that has not even been envisioned provides a directed approach to conceptualization, with some assurance that by fulfilling these requirements, the product will meet the needs, wants and desires of the customer and stakeholders, in this case the mobility impaired person that wants to “get around”. If desired by the user, the VOA LLM Bot welcomes any number of descriptions of competitive products or current state solutions; the user need only type such information into the Bot and let it know that it is a competitive product description. These competitive descriptions could be offerings found on the web, or descriptions of products from online retailers. In this case study, by describing three competitive services the number of product design requirements more than doubled, providing refined and targeted differentiation, such as adding in community and social features like shared rides to local community centres or events, integration with local senior activity schedules, and features to connect users with similar destinations.
5. Assessment of the use of the VOA LLM Bot
A pilot study was run to identify the potential value of the VOA LLM Bot. Carnegie Mellon University offers a Masters in Integrated Innovation for Products and Services (MIIPS) degree through its Integrated Innovation Institute. This professional degree cross trains engineers, marketers and designers in both overlapping and domain-specific skills needed to excel in product and service innovation. The Value Opportunity Analysis is taught and used by the students in at least two classes, the first being Integrated Product Development Methods which teaches the product innovation method, and the second in the capstone Integrated Product Development (IPD) course where interdisciplinary teams explore and provide product solutions for new opportunities for corporate sponsors. In addition to an in-residency program, an online version of MIIPS is offered for working professionals to complete on a part time basis. The online professional program used the VOA LLM Bot in the capstone IPD course. Because the students have previously learned and applied the VOA method in a prior class (and for many in practice in their jobs), they were an appropriate group from whom to understand the value of the VOA LLM Bot. There were only 6 students in this course section, and only 5 completed both surveys, one before being introduced to and using the VOA LLM Bot (but after having learned and applied the VOA method manually), and then after using the Bot in their class project; thus the data only includes 5 participants and the study is seen as preliminary. Comparisons were only sought to understand how the use of the AI agent changed participants’ perceived benefit and use of the VOA method.
A paired T test assessed that the designers found improvements of 1) the VOA as a useful method for identifying product requirements (p=0.03411, T=3.1623), 2) the VOA as a useful method to identify strengths and weaknesses in alternative opportunity solutions (p=0.03492, T=3.1379), and 3) understanding of how to use the VOA method (p=0.03492, T=3.1379) with significance; and improvement in finding the time investment of using the VOA appropriate for its purpose (p=0.0993, T=2.1318) with marginal significance.
Although the number of participants was small, they were each experienced professional product/service developers with strong affinity to the value of using the VOA LLM Bot. Future work will run a more extensive study on its use through various product development environments.
6. Discussion
Inherent in the Value Opportunity Analysis is the emotion that a product or service elicits in the user/stakeholder. Emotion is a fundamentally human attribute. Although we do not claim that the VOA LLM Bot is emotional, it is able to process and differentiate emotions that people feel, and to explore those feelings with the human users. The VOA LLM Bot effectively leads the human user through the VOA method, and also augments the knowledge of their stakeholders that the user brings to the interaction with the Bot. The result is a deep understanding of the needs, wants and desires of product stakeholders based on the research that the designer has uncovered, but also augmented by the vast knowledge base available to the LLM model. Most impressive is the depth and quality of the product design requirement description that results from use of the method, through an efficient process of application.
The VOA LLM Bot is available for the reader’s use: VOAiBOT.com
The VOA LLM Bot employs state of the art LLM technology to provide its service to the user. The work illustrates how LLM technologies can be tuned to provide benefit to designers to help understand and articulate the requirements for their design effort to more likely succeed. The work also illustrates how LLM technology can augment the design process, allowing for deeper and more efficient reasoning about design knowledge. Although a useful tool for design practice, the VOA LLM Bot also illustrates a role for augmenting teaching, aiding a student through a design process with direction, exploration, and meaning.
Of note, the capabilities of LLM technologies are expanding at a rapid pace. To maximize its effectiveness, the VOA LLM Bot may benefit from being updated at a regular cadence based on advances to LLM abilities.
Future work will also expand the scientific analysis of its effectiveness through more extensive studies of its use by design teams. Another focus of future effort is to explore human interpretation of the Bot’s mastery of human emotions.
Acknowledgement
The authors thank Peter Boatwright for his feedback on the VOA LLM Bot, and for incorporating it into his class.
Open access
