2. Visual-language models for engineering design

We performed a literature review of VL models within the context of ED applications. VL models can be divided into foundational models and fine-tuned models. Foundational models are trained on large, general datasets, so they can be applied across a wide range of tasks. Examples of foundational VL models are OpenAI’s GPT-4V (vision), DALL-E and Flamingo. These models are not specifically trained on design data. Fine-tuned models are created by adapting (finetuning) foundational models using domain-specific dataset, improving their performance on specialized tasks. Figure 1 maps key ED tasks across the ED phases: problem definition, conceptual, embodiment and detailed design. These ED tasks are further classified based on whether they have been addressed using foundational VL models, fine-tuned VL models, or remain as gaps in current research.

Figure 1. Visual-language models for engineering design tasks

Figure 1 shows that foundational VL models have been used for ED tasks across the conceptual, embodiment and detailed design phases. In the conceptual design phase, Edwards et al. Reference Edwards, Man and Ahmed(2024) used OpenAI GPT-4V (Vision) and DALL-E 3 to generate design concepts using sketches and textual descriptions. Picard et al. Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed(2023) used ChatGPT-4V and Llava 1.6 34B to assess design similarity of design sketches for concept selection. In the embodiment and detailed design phases, Picard et al. Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed(2023) used text and image prompts to perform tasks such as selecting materials, optimizing design topology, generating drawing descriptions, and developing CAD models. However, the effectiveness of these foundational VL models for precision-driven ED tasks remains limited. In fact, these models struggle to “understand” technical details, and their answers are never fully accurate (Edwards et al., Reference Edwards, Man and Ahmed2024). For instance, their current capabilities are not yet sufficient for interpreting complex technical drawings or producing viable CAD outputs (Picard et al., Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed2023). This limitation stem from the fact that foundational VL models have not been specifically trained on design data and their underlying design knowledge remains limited (Song et al., Reference Song, Zhou and Ahmed2024). As a result, they struggle to capture fine-grained design concepts.

To address this issue, several studies have fine-tuned foundational VL models using design-specific data. Yuan et al. Reference Yuan, Marion and Moghaddam(2022) developed a VL model to evaluate design concepts using product reviews with images and text. Similarly, Su et al. Reference Su, Song and Ahmed(2023) used 2,571 instances of online textual and visual information to develop a VL model for evaluating vehicle designs. Song et al. Reference Song, Miller and Ahmed(2023) fine-tuned a VL model to assess creativity of 1,086 freehand sketches with handwritten descriptions. Kwon et al. Reference Kwon, Huang and Goucher-Lambert(2022), using a dataset comprising 2D snapshots of 26,671 3D objects, finetuned a VL model capable of retrieving design stimuli by combining images with keywords describing component features in order to generate new design concepts.

As shown in Figure 1, fine-tuned VL models, have been used only in the conceptual design phase for design concept evaluation and generation. Furthermore, all previous approaches relied on small-scale design datasets for fine-tuning and employed simple images such as sketches rather than complex technical drawings. Moreover, the applications of fine-tuned VL models to other ED tasks, such as exploring existing solutions, identifying stakeholder needs, and generating technical drawings, remains unaddressed in the current literature due to the following main limitations.

(1) Multimodal Design Dataset Creation: the creation of large multimodal data for VL model finetuning is challenging because large accessible design datasets like Pinterest and Fusion 360 Gallery contain single modality data and manual labelling design data is time-consuming and resource-intensive (Song et al., Reference Song, Zhou and Ahmed2024, Jin et al., Reference Jin, Yang, Hu, Guo, Luo and Liu2024). Moreover, creating multimodal dataset involve aligning textual descriptions with corresponding images. This imply finding semantic relationships and correspondences between text and visuals. For example, given a technical drawing and a caption we must find areas of the drawing corresponding to the caption’s words or phrases. This is challenging because ED concepts like functions, components, unit of measurements and spatial orientations are often not explicitly represented in either the text or the visuals. Moreover, the relationships between modalities are often subjective (Baltrusaitis et al., Reference BaltruŠaitis, Ahuja and Morency2018). A single image can be described by multiple texts and vice versa, and a “correct description” of a technical drawing may not exist. For instance, Figure 2 demonstrates alternative textual and visual representations of a bearing-shaft assembly, where combinations of text-images pairs (T1-I1; T2-I2) convey the same information while complementing each other. As a results, there is a lack of high-quality multimodal datasets for ED applications. (Kwon et al., Reference Kwon, Huang and Goucher-Lambert2022; Picard et al., Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed2023; Song et al., Reference Song, Miller and Ahmed2023; Song et al., Reference Song, Zhou and Ahmed2024; Li et al., Reference Li, Wang and Sha2023; Consoloni et al., Reference Consoloni, Giordano and Fantoni2024, Jin et al., Reference Jin, Yang, Hu, Guo, Luo and Liu2024).

Figure 2. Two alternative uses of text and images to express functional and structural ED concepts

(2) Effective Representations of Design Knowledge: VL models are required to learn how to represent and join information from text and visuals in a way that exploits the complementarity and redundancy of the two modalities (Baltrusaitis et al., Reference BaltruŠaitis, Ahuja and Morency2018). In the context of ED, this is challenging and still far beyond an ideal level (Pan et al., Reference Pan, Li, Li, Hu and Bao2024), as it requires domain-specific understanding of how structural and functional design features are expressed across text and images (Kwon et al., Reference Kwon, Huang and Goucher-Lambert2022; Picard et al., Reference Picard, Edwards, Doris, Man, Giannone, Alam and Ahmed2023; Song et al., Reference Song, Miller and Ahmed2023; Song et al., Reference Song, Zhou and Ahmed2024). For example, Figure 2 illustrates how functional and structural information can be distributed between visual and textual descriptions when explaining a bearing-shaft assembly. Text segment T1 encodes functional information through phrases like “shaft rotates” and “withstands radial forces” while the corresponding image I1 conveys structural details, such as the shaft having a taper on the left-hand side using arrows to indicate dimensions. Conversely, an alternate case can occur, where text segment T2 describes structural information using terms like “coaxially mounted” and “20 mm taper”, whereas the corresponding image I2 represents functional information through visual elements such as arrows indicating rotation and force resistance.

3. The task: patent citations retrieval

In this work we develop a fine-tuned VL model for exploring existing solutions using a large-scale multimodal patent dataset consisting of 1.5M text-image pairs This task falls under the problem definition phase, and it has not been addressed with VL models in previous research.

In the field of ED, exploring existing solutions is a critical ED task. Patent databases are valuable repositories of technical knowledge and serve as a key source of information on prior art. When designing a new product, conducting effective prior art searches using patents enables designers to identify technical challenges, evaluate gaps in current solutions, and access a diverse array of design stimuli. However, this task poses significant challenges for engineers because it requires (1) capturing fine-grained technical similarities and differences between the proposed design and existing patented devices and, (2) a comprehensive understanding of patent text and drawings. Additionally, commercial patent search platforms rely solely on textual keyword searches, lacking image-based querying. This limitation affects prior art retrieval due to language ambiguity and terminology variations, often requiring iterative query expansion to accurately capture complex technological domains. For these reasons, VL models offer a promising solution not only to use both patent text and drawings to search patent database but also for facilitating cross-source retrieval by connecting patents with CAD models and online product reviews.

In this work, we finetuned the foundational VL model, developed by OpenAI, known as Contrastive Language-Image Pre-training (CLIP) model (Radford et al., Reference Radford, Kim, Hallacy, Ramesh, Goh, Agarwal and Sutskever2021), to perform prior art search on patent databases using both patent text and drawings. Specifically, we tested its performance on an extremely knowledge-intensive task: retrieving patent citations made by examiners. Patent citations are references to existing patents which are added by patent examiners during the examination process of patent applications. When an existing patent is cited, it indicates that examiners have assessed it as technically relevant to a pending application, potentially challenging the application’s patentability. By choosing this task, we aimed to evaluate our fine-tune VL model on a task where precision, high-level reasoning and domain-specific knowledge are critical.

5. Results and discussions

Table 2 presents the performance results for each retrieval task across k values of 1, 3, and 5, comparing the CLIP base model and our fine-tuned model. Bold numbers indicate the best-performing model for each column. The baseline model achieves its highest F1-score of 0.081 for k=5 on the joint task, indicating that, on average, it is not capable to accurately retrieve the 5 cited patents corresponding to each citing patent. In contrast, the fine-tuned model shows slight but consistent improvements across all tasks and k values, reaching a maximum F1-score of 0.09 at k=5 on the joint task. This indicates the contribution of our fine-tuning to very modest performance gains. However, the overall performance of both VL models remains insufficient for effective multimodal patent citation retrieval as their results are significantly below the optimal value of 1. This indicate that, our fine-tuned CLIP model produces embeddings that fails to capture technical details required for accurate patent citation retrieval within top-5 results.

Table 2. Results of patent citation retrieval tasks for CLIP base and fine-tuned model

To analyse the overall performance of our fine-tuned CLIP model, Figure 4 presents the plots of Avg. P@k, Avg. R@k, and Avg. F1@k metrics across k values up to 30 for the text-vs-image, image-vs-text and joint tasks. In all plots, as k increases, Avg. R@k increases because retrieving more patents naturally captures more cited patents. Conversely, Avg. P@k decreases with higher k because the inclusion of additional retrieved patents introduces more non-cited patents, reducing precision. Given that each citing patent has on average 5 cited patents, as expected, Avg. F1@k peaks around k=5 for all retrieval tasks. In fact, beyond k=5, while Avg. R@k continues to increase, Avg. P@k drops as more non-cited patents are retrieved, causing Avg. F1@k to decline.

Figure 4. Performance of retrieval strategies: Avg. P@k, R@k and F1@k for k values 1-30

Figure 4 indicates also that the joint retrieval task demonstrates relatively better performance compared to the text-vs-image and image-vs-text tasks, as reflected by higher F1-scores across all k values. Specifically, F1@k for the joint retrieval strategy lies within the range [0.05, 0.1], whereas for the other tasks, it remains within [0.0, 0.05]. This highlight that our fine-tuned CLIP model produces embeddings which do not enable cross-modal retrieval of patent citations (i.e., the model has limited ability to transfer ED knowledge between text and image modality) and, the joint embeddings demonstrate a slightly improved capacity to capture ED knowledge.

Previous results show that our fine-tuned CLIP model is unable to retrieve the cited patents with performance comparable to that of patent examiners. However, we evaluated the model’s capability to distinguish between patents from different IPC classes (i.e., technological domains) during retrieval. For example, given a citing patent which falls under the class A42B3, which relates to “helmets or other protective head coverings,” the model should avoid retrieving patents from unrelated classes, such as H02K19, which relates to “synchronous motors or generators”. Ideally, for a citing patent belonging to a specific IPC class, we expect the top 30 retrieved patents to belong to the same IPC class, even if they are not the exact cited patents. This expectation underscores the model’s ability to generate embeddings that capture general technological domain information, ensuring that retrieved patents remain at least relevant within the broader technological context of the citing patent.

Figure 5 shows the IPC class distribution among the top 30 retrieved patents for citing patents belonging to A42B3, A62B18, and H02K19 IPC classes. It helps to assess the model’s ability to maintain domain consistency by prioritizing patents from the same technological class. For example, in the case of class A42B3, 92% of retrieved patents belongs to the same class (A42B3), 7.22% are from A62B18, and 0.77% belong to class H02K19. For all IPC classes, the majority of the top-30 retrieved patents belong to the same IPC class as the citing patents. This shows that the model produces embeddings of first claims and images that can effectively distinguish between IPC classes during retrieval. However, for citing patents in A62B18, 19.21% of top-30 retrieved patents belong to class A42B3. This suggests that the model faces challenges in achieving fine-grained differentiation between closely related domains, such as A42B3 (helmets) and A62B18 (breathing masks). In contrast, it successfully distinguishes these classes from the significantly different technological domain of H02K19 (electric motors).

Figure 5. Distribution of IPC classes of retrieved patents for each citing patent

This indicates that despite the use of a large dataset for fine-tuning (1,5M text-image pairs), our model has primarily learned general ED knowledge sufficient for classify patents by IPC class. Notably, in literature patent retrieval has been done with higher accuracy using only text. A relevant example is the study by Siddharth et al. Reference Siddharth, Li and Luo(2022), which utilizes text-only embeddings for patent retrieval in ED applications. This rises a critical concern about multimodality: VL models are resource-intensive models in terms of energy, computing infrastructures and training time. Without effective multimodal representations to fully capture ED knowledge from both text and images, these models risk of being nothing more than oversized, inefficient classifiers.

6. Limitations and future research directions

This section examines the limitations of our methodology and proposes potential improvements for future research. These limitations are categorized into three layers: (1) Data, (2) Model, and (3) Testing, highlighting the specific areas where challenges occur.

The primary limitations related to (1) Data are as follows. (1.1) one limitation is using only the first claim as input, which relies heavily on technical-legal jargon and captures only surface-level details of inventions. Future research will incorporate additional patent sections, such as abstracts and detailed descriptions to provide a more comprehensive representation of technical details. (1.2) Patent first claims describe both functional and structural aspects of a device. This approach uses first claims as input without distinguishing these aspects. Classifying sentences into functional and structural categories using NLP methods could enable the generation of distinct functional and structural descriptions for patent images, enabling functional-structural indexing and retrieval of patent data (Song et al. Reference Song, Miller and Ahmed2023; Kwon et al. Reference Kwon, Huang and Goucher-Lambert2022).

The primary limitations related to (2) Model are as follows. (2.1) One key issue is that CLIP’s textual encoder truncates first claims to the initial 77 tokens, potentially omitting critical information required for patent citations retrieval. More precisely, first claims contain an average of 243 tokens (with a standard deviation of 152). As a result, the CLIP model discards approximately 70% of the information in first claims. To address this issue, as suggested by Lo et al. Reference Lo, Chu, Hsiang and Cho(2024), we plan to use LLMs to summarize functional and structural information of first claims using a prompt template. Additionally, to bypass CLIP’s text input limit, a hierarchical encoding approach could be employed. Rather than encoding the entire first claim at once, the text can be divided into smaller segments, each independently encoded to capture local semantic representations. Then, these individual representations can be aggregated into a single global embedding that capture the overall meaning of the first claim. (2.2) We implemented an early fusion approach, summing text and image embeddings element-wise to create a single joint representation. This is limiting because it treats both modalities equally and does not capture complex relationships between them. This could be improved by adopting more advanced multimodal fusion techniques, such as attention-based methods, which can capture deeper inter-modality relationships (Song et al., Reference Song, Miller and Ahmed2023; Li et al., Reference Li, Wang and Sha2023). (2.3) Pretrained VL models, such as our finetuned CLIP, operate as “black boxes” making it difficult to understand how they process images and text to produce outputs (BaltruŠaitis et al., Reference BaltruŠaitis, Ahuja and Morency2018). This lack of transparency limits the explainability and complicates the analysis of specific failure cases (Song et al., Reference Song, Zhou and Ahmed2024). For instance, it remains unclear which concepts of ED knowledge are represented in our embeddings, and which features of visual and textual inputs our model attends to (prioritize) when constructing the embedding space. As a result, quantitatively determining why some cited patents are retrieved while others are not remain extremely challenging and beyond the scope of this work. To address this, we plan to explore our embedding space using dimensionality reduction techniques such as PCA and t-SNE projection, which may provide macro-level insights into its structure (Lo et al., Reference Lo, Chu, Hsiang and Cho2024). Additionally, we also aim to use NLP techniques to identify ED concepts, including functions, components, material and spatial arrangements within first claims. By doing so, we aim to use these identified concepts as explanatory elements to interpret and debug misleading similarities in model’s outputs, facilitating the validation of model predictions against established ED knowledge (Song et al., Reference Song, Miller and Ahmed2023; Kwon et al., Reference Kwon, Huang and Goucher-Lambert2022). This approach will also contribute to refining the cosine similarity metric by re-weighting embeddings based on the importance of ED concepts, ensuring that similarity computations prioritize technical terms over common words.

The primary limitations related to (3) Testing are as follows. (3.1) Our methodology does not currently benchmark our VL model with a text-only model for patent citations retrieval. In future studies we plan to use an LLM to generate text embeddings of first claims and use them for our patent citations retrieval task. In fact, when working with VL models, evaluating whether the additional complexities of image processing and text-image alignment are justified by performance gains is crucial for advancing research in this field. (3.2) The CLIP base model is pre-trained on coloured RGB images, primarily composed of real-world photographs (natural images). Therefore, it may struggle to process black-and-white schematic images found in patents, as these differ significantly in style, structure, and complexity from the natural images used during its pre-training. To evaluate this pretraining biases, we plan to experiment with using existing VL models on natural images of technical objects, such as 2D snapshots of CAD models and online images of products. Since many technical images used by designers are black-and-white and schematic, this experiment will help us evaluate how well current VL models adapt to both natural and schematic technical images. (3.3) Our CLIP model is currently evaluated using patents from only three IPC classes (A42B3, A62B18, H02K19), limiting conclusions on model performance to these specific technological domains. For future studies, we plan to expand the dataset to include a broader range of IPC classes. Since the CLIP base model is primarily pre-trained on real-world photographs of everyday objects, it is likely to perform better on patents related to consumer products such as eyeglasses, chairs, and helmets, rather than on highly technical systems like solar batteries, harvesting machines, or industrial furnaces. Conducting further experiments will provide deeper insights into how effectively existing VL models can be fine-tuned for specific technological fields and design challenges. (3.4) Our approach relies on examiner citations to automatically build a database of linked patents, eliminating the need for time-consuming manual relevance evaluations. While this method is effective for evaluating the model’s ability to retrieve relevant patents, it does not account for the fact that patent citations are backward-looking (i.e., a cited patent must always be older than the citing patent). As a result, our VL model may retrieve subsequent relevant patents. In future studies we plan to adjust the retrieval metrics to ensuring that only prior art is considered for retrieval (Luo et al., Reference Jin, Yang, Hu, Guo, Luo and Liu2024).