1. Introduction
The emergence of generative artificial intelligence (AI), such as the large language models (LLMs) and visual language models (VLMs), has brought significant impacts to diverse areas, including engineering design. In recent years, researchers and designers have gained sufficient interest to discover the next era of design by bringing massive and detailed applications of AI during the whole design process to achieve human-AI collaborative design (Reference LiuLiu, 2025). Rather than functioning merely as passive tools, AI systems are increasingly expected to serve as cognitive partners that actively contribute to design activities—providing inspiration, enriching partial concepts, and even translating early-stage ideas into feasible prototypes (Reference Viros-i-Martin and SelvaViros-i-Martin & Selva, 2021). As AI-assisted applications continue to advance in intelligence and diversity, the product design process is expected to become substantially more streamlined, reducing both cognitive and operational costs for designers. This shift will lower the barrier and encourage more participants into the design process with the assistance of AI tools. Achieving this would requires not only advances in AI capabilities but also reconsider how humans should interact with AI-empowered computing systems (Reference Wang, Churchill, Maes, Fan, Shneiderman, Shi and WangWang et al., 2020). In fact, although AI tools can accelerate and simplify the design process, how to use such tools effectively so that designers can genuinely benefit from them remains an open concern. For instance, Reference Wadinambiarachchi, Kelly, Pareek, Zhou and VellosoWadinambiarachchi (2024) conducted a case study in which designers incorporated AI-generated images during the ideation process. Their results showed that designers often fixated on the provided images, generating ideas with reduced variety and lower originality. Balancing the use of AI tools in design would be critical (Reference Cheng, Chen, Song, Zhang, Li and SunCheng et al., 2025). Designers need to ensure they can integrate them into their workflow to enhance efficiency, while avoiding the abuse of them, which can potentially impact design creativity. In product design, multiple intermediate steps are often required between the initial idea and the final concrete presentation. Designers usually need to prepare multiple potential design solutions and gradually clarify the design direction by filling in sufficient implicit details. Although various generative AI-based methods have emerged to assist designers by adding details and visualize representations of designs. Despite recent achievements, state-of-the-art generative AI models still face substantial challenges in explainability, and their responses often lack reliability and robustness (Reference Mavrepis, Makridis, Fatouros, Koukos, Separdani and KyriazisMavrepis et al., 2024). Meanwhile, these models still struggle to perceive latent user intentions or to interpret vague and abstract concepts effectively. As a simple yet effective optimization strategy, a well-crafted prompt can significantly improve the reliability of the model to generate outputs that align with user expectations (Reference Ye, Ahmed, Pryzant and KhaniYe et al., 2024). However, as a new and emerging techniques in generative AI, researchers and designers may not familiar or not professional in prompt engineering (Reference Sahoo, Singh, Saha, Jain, Mondal and ChadhaSahoo et al., 2024). In design aspect, for example, designers would need to carefully construct the prompt before integrating the AI tools into their workflow. They would need to add sufficient details to guide the model and assess the responses to optimize the prompt before achieving the expected results. Therefore, developing a workflow that can integrate with AI tools for assisting designers in implementing and elaborating an idea from scratch while providing feedback across multiple modalities would be highly beneficial. Although the increasing use of generative AI has brought significant changes to early-stage design, there remains a lack of systematic frameworks to support the elaboration of vague design ideas into coherent and controllable representations. This gap leads to the following research question: How can multimodal language models be structured to support the elaboration of early-stage design ideas while maintaining designer control and improving the reliability of visual feedback?
In this work, we propose a multimodal language model-based design assistance framework for the early stages of product design. It presents a complete workflow from idea parsing and semantic refinement to the generation of structured image instructions. The framework leverages LLMs to perceive designer intentions in the early-stage ideas, enriches the idea with necessary details, and generates structured instructions for VLMs to improve the accuracy and consistency of visual feedback. In detail, we introduced a mood board into the workflow to enhance the relevance of feedback during the enrichment of design details. Additionally, we designed an interactive stage where designers can select, optimize, or customize idea details and generate corresponding instructions for visual support. To evaluate the effectiveness of the proposed framework, we conducted a study using ten initial ideas collected from designers and generated corresponding image instructions. Compared with direct image feedback, the proposed method can add sufficient details to the ideas while enabling designers to adjust instructions and maintain the reliability of the results. To the best of our knowledge, this is one of the first work that explore human–AI collaboration during the ideation stage of design, providing more effective support for designers in concept exploration and visual expression.
2. Related works
AI has been explored and adopted for supporting a wide range of design activities, particularly during the early phases of design. Conceptual design, including both idea generation and evaluation, is a key area of focus. Prior studies have employed image-generative AI models, such as Midjourney, Generative Adversarial Networks (GANs), DALL·E, in supporting design idea generation, for example, by producing visual design stimuli through synthesising product and object images (Reference Chen, Sun and HanChen et al., 2023; Reference Chen, Wang, Dong, Shi, Han, Guo, Childs, Xiao and WuChen et al., 2019; Reference Wang and HanWang & Han, 2023). Text-generative AI models, such as LLMs, have been increasingly adopted recently to generate design ideas to facilitate conceptual design by employing design methods and approaches such as analogical reasoning, bio-inspired design, TRIZ, and sustainable design (Reference Chen, Cai, Jiang, Luo, Sun, Childs and ZuoChen, Cai, et al., 2024; P. Reference Jiang, Han and Ahmed-KristensenJiang et al., 2025; S. Reference Jiang, Han and Ahmed-KristensenJiang et al., 2025; Reference Wang, Zhao, Zuo, Song, Han, Childs and ChenWang et al., 2025; Reference Zhu, Zhang and LuoZhu et al., 2023). Several studies have shown the application of both image- and text-generative models to further support designers in creative idea generation (Reference Chen, Zhang, Han, Sun, Childs and WangChen, Zhang, et al., 2024; Reference Li, Chen, Wu, Yao, Zhang, Liu and SunLi et al., 2025; Reference Obieke, Milisavljevic-Syed, Jiang, Bridgeman and HanObieke et al., 2025). In addition, Reference Han, Obieke, Zhao and JiangHan et al. (2024) and Reference Jin, Yang, Hu, Guo, Luo and LiuJin et al. (2025) have demonstrated the potential of video-generative models, such as Sora and enhanced stable diffusion models, for producing dynamic video stimuli. These models offer insights into aspects like motion and mechanisms that static image- or text-based generative models cannot capture, thereby enriching creative design idea generation. Building on the semantic understanding, reasoning, instruction-following, and natural language generation abilities of LLMs, recent studies have explored their application in idea evaluation. Using techniques such as Chain-of-Thought (CoT) prompting and multi-agent architecture to improve LLMs’ reasoning and reliability, and adopting existing idea evaluation frameworks, studies have shown that LLM-driven design evaluations align closely with assessments made by human design experts (Reference Zhang, Han and Ahmed-KristensenZhang et al., 2025a, Reference Zhang, Han and Ahmed-Kristensen2025b). In addition to conceptual design, studies on the use of AI techniques in design have explored various applications. For example, leveraging LLMs to create personas (Reference Schuller, Janssen, Blumenröther, Probst, Schmidt and KumarSchuller et al., 2024; Reference Zhou, Fang, Ding, Cheng, Yan, Zhu, Bao, Wang and SongZhou et al., 2024), adopting image-generative models, such as Vega AI and Midjourney, to develop design forms (Reference Lu, Hsiao, Tang and WuLu et al., 2024), applying LLMs to assist the design flow through established design method cards (Reference Chen, Cheang, Jiang, Xu, Cai, Sun, Childs, Han, Hansen and ZuoChen et al., 2025), and enhancing user requirement management by integrating LLMs with knowledge graphs (Reference Liang, Wang, Li and YanLiang et al., 2024).
3. Methodology
3.1. Idea detail enrichments with LLMs
While using LLMs to support detail enrichment in design tasks, we noticed that the models could exhibit low consistency and may fail to provide reasonable or relevant details. One possible explanation is that producing more diverse or unexpected outputs with a higher temperature setting can increases the variability of responses, which can also reduce the stability of the generated details. Although higher sampling temperatures are commonly believed to encourage creativity, Reference Peeperkorn, Kouwenhoven, Brown and JordanousPeeperkorn et al. (2024) suggest that temperature does not reliably control creativity and has only a limited influence on novelty. Instead, higher temperatures primarily increase randomness, which can undermine coherence and consistency. Therefore, a more feasible approach is to support the model by providing richer contextual information, so that it can better understand the design ideas and tasks, leading to more stable and robust outputs. As an important interactive tool in design practice, mood boards facilitate the articulation of aesthetic direction and contextual elements, supporting designers during the early stages of the design process (Reference LuceroLucero, 2012). The mood board can also provide effective feedback by visualizing abstract concepts, helping designers refine their ideas, add details, and maintain overall consistency in design style (Reference Munk, Sørensen and LaursenMunk et al., 2020). Inspired by the use of mood boards in engineering design, we adopt a similar mechanism into our LLM-based detail enrichment process to provide structured guidance while maintaining relevance in the generated outputs. Specifically, we enhance the model by supplementing the initial innovative ideas proposed by designers through three aspects: target user, environment, and mood. We will first present the following three questions to the LLMs and ask the model to answer the questions accordingly:
-
1. Target users: Who will use the product? What are their age group, background, and lifestyle?
-
2. Mood: What mood should the product convey? What kinds of forms, colours, and materials would help express that mood?
-
3. Environment: Where this idea is likely to be used?
The LLMs are asked to assess the design idea and provide detailed answers to these questions. Through this process, the model gains a more concrete understanding of the target users, application environment, and intended mood of the idea. With this contextual information, it can then supplement the designer’s initial concept with more appropriate, relevant, and coherent details.
3.2. Multimodal language models for precise visual feedback generation
After the LLM completes the detail enrichment process, the design idea and details will be organized into a structured textual representation. While this semantic enrichment improves conceptual clarity, designers would still need visual feedback to evaluate and further develop their ideas. Therefore, the structured textual results still need to be transferred into a carefully designed instruction format that can guide VLMs in generating more precise and coherent visual feedback. As presented in Figure 1, after receiving the detailed analysis of the target users, mood, and environment, we then ask LLMs to further expand the design concept and translate these enriched details into coherent and actionable image-generation instructions. We ask the model to provide external supplementary information so that the generated visual feedback can be more precise and controllable. In this work, we ask the models to provide external details across the following parameters.
-
1. Background: A brief description of the surrounding environment in the generated image.
-
2. Application scene: The overall context or situation depicted in the image.
-
3. Shape/Form: Geometric features that define the physical form of the product’s appearance.
-
4. Materials: The specific materials or material combinations intended for the product.
-
5. Color palette: The primary colors or tonal scheme used in the product design.
-
6. Texture: The surface qualities or material characteristics that will be represented in the product.
-
7. Viewpoint: The camera angle or perspective from which the product is shown.
-
8. Lighting: The type and direction of illumination used in the image.
-
9. Render type: The visual style or rendering method to be applied. In our framework, the default is a realistic rendering style.
We ask the model to provide brief descriptions of these parameters and then integrate the details with the structured textual content to image generation instructions. However, LLMs do not always provide image details that can align with the expectations of designers. For instance, when using VLMs to generate sample images of the product, some designers may prefer to place the product within a concrete application scenario, whereas others may opt for a clean background to focus on the product presentation. Any parameter that influences image generation should be considered and explicitly defined to ensure that the resulting image can meet the intended requirements. To address the aforementioned issue, this work presents an interactive process that enables designers to specify or refine visual parameters. As shown in Figure 1, during the design detail enrichment stage, we ask the LLM to generate multiple reasonable options for the parameters listed above. Designers can then select, refine, or input textual content to customize the generated image.
Once designers have configured the above parameters, we use the LLM to reconstruct the structured details into a coherent textual instruction, which is then passed to the VLM to generate the corresponding visual output for the design idea.
Detailed workflow of the proposed method
Figure 1 Long description
The image contains one illustration and one diagram. The illustration shows a conversation between a user and a bot. The user mentions having a design task and an idea, asking the bot to expand details. The bot responds by asking the user to answer mood board questions and provide structured instructions for image generation. The diagram shows a form for selecting design parameters, including background, shape/form, application scene, and materials. The form is filled with options such as clean studio white for background, double-walled tapered cylinder for shape/form, rush-hour subway commute for application scene, and matte stainless steel body for materials. The diagram also includes images of a wide-bottom mug in different settings, illustrating the design idea of a wide base to help with stability.
4. Experiment and results
4.1. Experiment settings
To verify the effectiveness of our approach, we conducted a case study using the proposed method to process and extend initial ideas, and to examine whether the reconstructed instructions lead to more coherent and visually informative concept imagery. We organized a design session in which participants were asked to generate novel ideas for two specific design tasks:
Design as many novel ideas as possible that hold hot liquid for drinking?
Design as many novel ideas as possible to collect and store rainwater for watering gardens?
We asked participants to propose novel ideas for the above design tasks based on their own interpretation of the requirements. To ensure diversity in perspectives, we recruited participants from a range of backgrounds. Some participants had substantial design experience, including professional designers and design researchers, while others were recruited from the general population and did not possess formal design training. While processing the collected ideas, we noticed that only a subset of ideas contained explicit design details. In contrast, most ideas consisted of short phrases, single words, or incomplete sentences. For example, in the first task, which asked participants to propose a product for holding hot liquid for drinking, numerous responses were brief and generic, such as “cup,” “spoon,” or “bowl,” which provide limited design specificity or creativity. As a matter of fact, this also inspired us to conduct this work. We use LLMs to enrich these initial ideas with details and leverage VLMs to generate corresponding product concept images that provide visual feedback for designers. In this study, we randomly sampled ten ideas from participants who were unfamiliar with design: five ideas related to products that can hold hot liquid for drinking, and five ideas focused on collecting and storing rainwater for garden use. Here we present the ideas sampled from this session, as summarized in Table 1.
Ideas used in this test for detail enrichment and image generation
In this test, to evaluate the contribution of our method, we compare three different input conditions for image generation.
-
• Original ideas (baseline): In this condition, the raw idea text provided by participants, together with the task description, was directly submitted to DALL·E 3 for image generation.
-
• LLM-enriched instructions: In this condition, we used GPT-5 to answer the guiding questions regarding target users, usage environments, and intended mood for each product idea. The model produced a structured analysis, which was then used in a second GPT-5 session to integrate the enriched details and formulate image-generation instructions for DALL·E 3. To encourage variation in the visual outputs, we set the sampling temperature to 1.0 and repeated the generation process three times for each idea, then generated the corresponding images.
-
• Designer-guided instructions: In this condition, we used the same structured analysis generated by GPT-5. Instead of directly producing a single detailed specification for the parameters listed in Section 3.2, we asked the model to generate five relevant options for each parameter, allowing designers to review, adjust, or customize the settings for image generation (e.g., background, scene, materials, colour palette, etc.). After designers finalized the desired image details, the refined parameters were sent back to GPT-5 to reconstruct a complete image-generation instruction, which was then submitted to DALL·E 3. As in the other conditions, we set the sampling temperature of GPT-5 to 1.0 and generated three image samples for each idea to ensure consistency. Additionally, designers were asked to create two distinct image configurations for each idea, and images were generated for both settings to support further comparison.
5. Experiment results
As shown in Table 2, we present the images generated from the original ideas and compare them with the images with detail augmentation using GPT-5. The augmented results demonstrate more stability and better alignment with the initial design ideas. All images with LLM-enriched instructions were generated using a realistic rendering style, whereas the images produced directly from the raw text showed higher variability, including realistic and hand-drawn sketch-like appearances. Moreover, the images generated with enriched instructions can demonstrate the intended product functions. For instance, images for idea_2 and idea_5 include steam to reflect their purpose as containers for hot liquid. While images for idea_6 display a desert environment, which aligns with the original description, as it is designed for areas with little rainfall.
Images generated with original ideas and LLM-enriched instructions
Table 2 Long description
The table contains two main columns: Image generated with original text and Image generated with LLM-enriched instructions. Each row represents a different idea ID from idea_1 to idea_10, with corresponding design tasks. Panel A: Image generated with original text. This column shows images created directly from raw text, displaying higher variability including realistic and hand-drawn sketch-like appearances. Panel B: Image generated with LLM-enriched instructions. This column shows images generated using a realistic rendering style, demonstrating more stability and better alignment with the initial design ideas. The images with enriched instructions also reflect intended product functions, such as steam for containers for hot liquid and a desert environment for areas with little rainfall.
Table 3 presents the images generated from designer-configured instructions. In this table, we list the parameters that designers selected or customized, along with the keywords extracted from the corresponding image-generation prompts. Comparing the images in Tables 3 and 2, the designer-configured results exhibit better consistency. Unlike the images in Table 2, which sometimes varied in style or composition, images in Table 3 can strictly emphasize the product itself and preserve a clear, isolated presentation of the main subject, which also aligned with the expectations for product concept imagery generations. The images also follow the specified parameters by the designers. For example, the background defined in idea_2 as “snowy window scene, frosted glass” is well presented across all generated samples. Furthermore, details from the original ideas are preserved. For instance, images for idea_5 include a lock or buckle on the lid, as it is described as “secure lid” in the initial idea.
Images generated with instructions using designer-guided details
In addition, to further test the stability of the proposed method, we conducted an additional test using the five ideas from Task 1. We adjusted the background parameter and set it to an application in office environments while keeping the remaining parameters mainly unchanged. As shown in Table 4, the generated images remained highly stable as the product forms were preserved. The background was updated to an office environment and added elements such as monitors and laptops to reinforce the scenario. These images suggest that designers using the proposed method can adjust specific parameters without compromising the coherence of the product concept imagery. It allows designers to refine the generated outputs for precise visual feedback.
Images generated with different settings of the background and application scenes
To conduct a systematic assessment of the proposed method and evaluate the overall quality of the generated images, three experts with over ten years of professional design experience were invited to participate in the evaluation. The experts assessed whether the generated images were able to fulfil the given design tasks. Two evaluation questions were used: “Q1. How well do the generated images match the textual idea?” and “Q2. How well do the generated images address the design task?”
Averaged expert evaluation results with standard deviation (SD)
Table 5 presents the details of expert evaluation results for generated images. In this evaluation session, all judgments were recorded using a 7-point Likert scale, from 0 (Does not match/address) to 6 (Excellent). We collected scores for idea images generated with different instruction strategies and presented the average and standard deviation to show how well the images perform on these two questions. In this test, the average scores for images generated with designer-guided details are 4.08 (Good) on the idea matchiness evaluation (Q1) and 4.36 (Good) on the task fulfilment evaluation (Q2). The results indicate that using designer-guided details can improve the overall quality of the generated images. Expert evaluation quantifies the value of images across two aspects, thereby verifying the effectiveness of the method in improving the image quality generated.
6. Discussion
In this work, we introduced a multimodal design assistance tool for early-stage ideation, leveraging LLMs to expand initial design ideas and support interaction with VLMs for visual feedback generation. Although the proposed framework can be effective in supporting designers during the ideation stage, several limitations remain. First, this work relied on GPT-5 to expand design details. Although most generated content was reasonable and aligned with design intent, some suggestions were tangential or irrelevant. Even when multiple options were provided, such outputs could still influence intensions of designers and potentially constrain creativity. Future work will focus on improving the detail-expansion stage to support novel and diverse design directions better. Second, while the proposed method significantly improved image quality for the hot-liquid design task, the improvements were less pronounced for the rainwater-collection task, as shown in Table 3. This may be attributed to the greater systemic complexity of rainwater-related products, which makes it more challenging for current VLMs to capture design intent and generate appropriate visual representations accurately. Future work will aim to improve the robustness of the image generation stage and extend the approach to service design scenarios. Finally, the current study remains at the research-prototype stage, with only a limited number of design ideas evaluated. In future work, we plan to scale this framework into a more systematic and deployable tool that can be applied in real-world design environments to support sustained human–AI collaboration.
7. Conclusion
This work introduced a novel framework for leveraging AI tools to support the ideation stage of design. We used LLMs to help designers enrich the details of early-stage design ideas and developed an interactive system that allows designers to refine and customise idea details. In addition, we proposed an autonomous instruction-generation process that transforms enriched design information into precise prompts for VLMs, enabling accurate and coherent visual feedback. To evaluate the effectiveness of the framework, we collected and tested initial product design ideas. The results demonstrate that the proposed method produces highly reliable and consistent images. The method also presents great scalability that allows designers to configure and refine visual parameters to pursue more expected and controllable image representations.
Acknowledgement
This work is funded by DIGITLab, UKRI Next Staged Digital Economy Centre (EP/T022566/1).
Open access
