Policy Significance Statement

This study demonstrates that social, cultural, and political (SCP) symbols embedded in Instagram posts play a significant role in shaping user engagement, perceptions of fairness, and the spread of information during misinformation campaigns. Through the integration of semiotic analysis, large language models, and epidemiological modeling, we find that symbol-rich content, particularly cultural references, contributes to greater trust and faster dissemination of messages. These findings offer valuable insights for social media governance by helping policymakers and platform designers enhance moderation systems and develop more effective strategies to counter misinformation. Emphasizing symbolic awareness in public communication and algorithmic audits provides a practical approach to strengthening digital resilience, supporting democratic discourse, and safeguarding information integrity during elections and other critical civic moments.

2. Background

Taiwan’s 2024 election encountered substantial misinformation challenges, including false allegations of voter fraud and miscounted results. As ballots were tallied on January 13, rumors of vote fabrication spread, raising concerns about election integrity. However, Taiwan successfully mitigated a potential crisis through a “whole-of-society” approach. Fact-checking organizations swiftly debunked false claims, the Central Election Commission clarified discrepancies, and YouTube influencers rapidly countered misinformation. This coordinated response helped safeguard election integrity and restore public confidence.

Figure 1, based on publicly available articles (Klepper and Wu Reference Klepper and Wu2024; MyGoPen 2024), illustrates the timeline of these actions, with misinformation marked in orange and debunking efforts in blue. While early research efforts focused on YouTube and TikTok, Instagram emerged as the primary source of anti-misinformation data for this study due to the large volume of relevant content available.

Figure 1. Timeline depicting the spread and rapid debunking of election-related misinformation in Taiwan. Orange depicts the misinformation campaign while blue depicts the anti-misinformation campaign.

3. Literature review

Semiotics, the study of signs and symbols in communication, provides a foundational lens for analyzing how meaning is constructed and interpreted through visual media. As Chandler (Reference Chandler2002) explains, images operate on two primary levels: the denotative (literal meaning) and the connotative (associative meaning). Symbols embedded within images can evoke a wide range of emotions and cultural associations, often extending far beyond their surface representation. For instance, a dove may represent peace or spiritual freedom depending on its cultural context.

In the digital era, especially on image-centric platforms like Instagram, the symbolic power of visuals has intensified. Van Leeuwen (Reference Van Leeuwen2005) argues that social media amplifies the connotative force of images, while Rose (Reference Rose2022) emphasizes the role of multimodality, noting that political symbols are interpreted through the viewer’s ideological and cultural filters. These interpretations can reinforce existing beliefs and shape broader political narratives. This aligns with Hall’s encoding/decoding theory, which posits that media messages are encoded with preferred meanings by content creators but are actively interpreted by audiences, who may accept, negotiate, or oppose those meanings depending on their cultural context Hall (Reference Hall2014).

Symbols used in social media often have deep historical and cultural roots. Social symbols such as gestures or rituals communicate shared meanings and contribute to social cohesion Turner (Reference Turner1967). Cultural symbols, including flags and religious icons, encapsulate collective identity and values, serving as markers of tradition and belonging Dadze-Arthur (Reference Dadze-Arthur2017). Political symbols, such as slogans and emblems, express ideologies, authority, and mobilization efforts Mosse (Reference Mosse2023). In today’s digital landscape, these symbols are dynamic and fluid, frequently repurposed or contested in real time. They play a pivotal role in identity formation, public discourse, and collective action. As Johann (Reference Johann2022) notes, symbols in the digital age often transform into viral content or memes, influencing public opinion and driving sociopolitical change. Consequently, the need for computational analysis of these symbolic elements has become increasingly urgent.

Entity extraction from images is a key step in this analytical process. Traditionally, this task has been addressed through computer vision techniques such as object detection and classification using convolutional neural networks (CNNs). Early architectures like AlexNet, VGGNet, and ResNet laid the groundwork for extracting visual patterns and hierarchies Ren et al. (Reference Ren, He, Girshick and Sun2015). More recently, the emergence of large language models (LLMs) has significantly advanced this field. Unlike conventional models, LLMs such as GPT-4 can process multimodal inputs and generate semantically rich outputs. They not only enhance contextual understanding but also offer detailed visual descriptions and entity recognition capabilities OpenAI (2024).

LLMs have transformed artificial intelligence by enabling high-accuracy text generation and comprehension across diverse contexts. Models like GPT-3 and GPT-4, trained on extensive datasets, excel at identifying linguistic patterns and semantic nuances Brown et al. (Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever and Amodei2020). Their applications extend to data annotation, autonomous agents, and content moderation. In annotation tasks, LLMs can consistently label large datasets with minimal human oversight, improving scalability and reducing errors Cui et al. (Reference Cui, Zhao, Liang, Li and Shao2022). For visual data, multimodal models such as CLIP (Contrastive Language-Image Pre-training) and Gemini bridge textual and visual domains. CLIP enables alignment between image content and language descriptions, facilitating tasks like captioning and symbolic recognition Radford et al. (Reference Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin and Clark2021). Gemini extends this functionality across audio, video, and text, offering a robust solution for multimodal reasoning and analysis Anil et al. (Reference Anil, Borgeaud, Wu, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth and Millican2023). In addition to symbol detection, it is essential to understand how symbolic content spreads across digital networks. To this end, researchers have adapted epidemiological models, originally developed to study infectious diseases, for use in analyzing the spread of information and misinformation on social media. The SIS (Susceptible-Infected-Susceptible) model captures recurring user engagement patterns, while SEIR (Susceptible-Exposed-Infected-Recovered) introduces a latency phase to represent delayed user action Daley and Kendall (Reference Daley and Kendall1964); Moreno et al. (Reference Moreno, Pastor-Satorras and Vespignani2002); Lerman and Ghosh (Reference Lerman and Ghosh2010). The SEIRS model adds resusceptibility, modeling the reactivation of users Zhao et al. (Reference Zhao, Wang, Cheng, Chen, Wang and Huang2011); Vosoughi et al. (Reference Vosoughi, Roy and Aral2018). Among these, the (Susceptible-Exposed-Infected-Skeptic (SEIZ) model offers particular relevance for narrative propagation. As shown by Gurung et al. (Reference Gurung, Agarwal and Al-Taweel2024a), SEIZ effectively captures how users may engage with, resist, or reject information. This makes it especially suitable for modeling misinformation spread, where skepticism plays a key role in moderating engagement.

4. Methodology

The following section outlines the data collection process on Instagram and explains the methodology behind the experiments. It highlights the extraction of social, political, and cultural (SCP) symbols and evaluates the performance of different large language models in this context. Finally, we explore key experiments, including modeling information spread and comparing factors such as trust and emotion, that provide insights into our approach and findings.

4.1. Data collection

In this section, we outline the methodology for collecting data on Taiwan’s election anti-misinformation campaign from Instagram. Our data collection followed a multi-stage, iterative approach designed to comprehensively capture information on the election and the related anti-misinformation efforts. The process began with identifying key seed terms based on the news sources discussed in Section 2.

Next, we constructed a co-occurrence network and performed topic clustering using Latent Dirichlet Allocation (LDA), as illustrated in Figures 2 and 3, from the seed posts collected. This analysis identified distinct communities centered around political figures (e.g., Lai Ching-te, Takaka Kiyoshi) and political entities (e.g., the Democratic Progressive Party (DPP) and Kuomintang). Insights from this network analysis and topic clustering guided the development of a refined keyword list incorporating terms related to Taiwan’s anti-misinformation campaign.

Figure 2. Topic Clustering using LDA showing keywords.

Figure 3. Communication network showing communities formed using keywords and hashtags.

Using these expanded keywords, we conducted a second round of data collection. To ensure comprehensive coverage, we employed a snowball sampling approach, allowing the dataset to grow dynamically as new relevant hashtags and topics emerged. The final set of hashtags and keywords is detailed in Table 1.

Table 1. Hashtags and keywords used for data collection from January 13th to 27th, 2024

4.2. Extraction of social, cultural, and political symbols

Large language models (LLMs) have been widely applied across various domains, including healthcare and education. One of their key strengths lies in their ability to understand context, significantly enhancing Natural Language Understanding (NLU) capabilities, as noted by Liu et al. (Reference Liu, Zheng, Du, Ding, Qian, Yang and Tang2023). Additionally, Qi et al. (Reference Qi, Fang, Zhang, Sun, Wu, Liu, Lin, Wang and Zhao2023) highlights the multimodal capabilities of these models, emphasizing distinct advantages: GPT-4 excels in delivering precise and concise responses, while Gemini is adept at generating comprehensive, richly detailed answers supplemented with relevant imagery and links. Recent advancements, such as Gemini Pro-Vision and GPT-4o, have further expanded these capabilities to include image analysis. In our study, we leveraged these advanced models to extract social, cultural, and political (SCP) symbols from images.

We briefly discuss the parameters used by LLMs to extract these symbols. Despite variations in overall size, GPT-4o and Gemini Pro-Vision share several key parameter settings. Below, we highlight the parameters consistently applied to both models to ensure uniform performance. It is important to note that these represent only basic tuning, as the primary focus of this paper is not on how these models compare in symbol extraction. Instead, we aim to observe how these models, in their standard configurations, approach the task of extracting these symbols.

• • Temperature: This parameter controls the randomness of responses. A lower temperature produces more predictable outputs, while a higher value increases variability. For both models, the temperature was set to 0 to prioritize predictability and consistency.

• • Frequency Penalty: This setting reduces the likelihood of repeated words or phrases by penalizing frequent tokens. In both models, the frequency penalty was set to 0, allowing for a natural language flow without discouraging necessary repetition.

The prompt employed for both models was the following:

“Visualize yourself as a proficient linguist tasked with analyzing an Instagram image. Your goal is to identify and categorize three forms of ‘Symbolic Communication’ present in the image: ‘Social,’ ‘Cultural,’ and ‘Political.’ If a classification is unclear, assign it a value of ‘0.’ For instance, if ‘Social’ and ‘Cultural’ are present but ‘Political’ is not, your output should be formatted as: {‘Social’: 1, ‘Cultural’: 1, ‘Political’: 0}. Justify each assigned value with reasoning.”

Figure 4 showcases examples of various combinations of social, cultural, and political (SCP) symbols as interpreted by the Gemini and GPT models. This comparison highlights both the alignments and divergences in how the two models evaluate and classify these symbols within the given context.

Figure 4. Exploration of images comparing various combinations of social, cultural, and political (SCP) symbols and their performance in GPT-4o and Gemini Pro-Vision.

In the first row of images, the models largely agree on their identification of most symbols, with one notable exception: the interpretation of a flower in the first image. Gemini attributes cultural significance to the flower, identifying it as a meaningful symbol, while GPT does not assign it any specific symbolic meaning, highlighting subtle differences in how the models prioritize cultural context and aesthetic elements. The flower’s vibrant variety of colors and species reflects the harmony Taiwan seeks to cultivate within its society and environment, making it particularly relevant to our case study on an anti-election misinformation campaign, as it symbolizes unity, trust, and the public’s confidence in the electoral process and commitment to peace.

In the second row, the disparities between the models become more evident. A clear example is the second image in this row, which features a dragon. GPT identifies the dragon as a cultural symbol, likely drawing on its traditional and historical associations, whereas Gemini does not classify it as such. Furthermore, the same image reveals differences in how the models interpret political symbolism. According to GPT, the dragon’s menacing posture toward an individual holding the Taiwanese flag reflects political tension, symbolizing the strained relationship between China (represented by the dragon) and Taiwan (represented by the flag-bearer). In contrast, Gemini interprets the image as Taiwan’s assertion of sovereignty and national identity in the face of perceived dominance or aggression from China. While both models recognize political undertones, their interpretations emphasize different aspects: GPT focuses on the antagonistic visual elements, while Gemini highlights the symbolic representation of resistance and identity.

The most significant divergence occurs in the interpretation of the final image. GPT categorizes this image as encompassing all three types of symbols—social, cultural, and political. According to GPT, the image conveys social unity through the large gathering of people, cultural pride through the traditional illuminated structure, and political expression through the presence of flags and banners and through the act of public assembly. In contrast, Gemini does not identify any symbols in this image. This stark difference underscores the models’ varying thresholds for detecting and classifying complex, multi-layered symbolic elements, especially in intricate visual compositions.

Additionally, we extended our analysis to examine the expression of trust and emotion within the comments on these posts, leveraging models such as GPT-4 and Gemini. Unlike the SCP extraction process, this analysis was performed without any parameter tuning, ensuring that the outcomes reflected the models’ default configurations. This approach allowed us to evaluate the models’ inherent ability to interpret and respond to emotional cues and trust indicators embedded in the data.

4.3. Categorization by symbol number

The images were categorized based on the diversity of symbols present in each post, without allowing for overlap (i.e., mutually exclusive):

• • Category 0: Posts with no symbols.

• • Category 1: Posts containing a single symbol of any type—Social (S), Cultural (C), or Political (P).

• • Category 2: Posts containing symbols from two of these types.

• • Category 3: Posts containing symbols from all three types.

The distribution of posts categorized based on symbol diversity is shown in Figure 5.

Figure 5. Data distribution of Instagram posts related to Taiwan’s anti-election misinformation campaign.

4.4. Categorization by symbol type

The images were categorized based on the presence of each symbol type, allowing for overlap (i.e., mutually inclusive):

• • Images with no symbol

• • Images with social symbol

• • Images with cultural symbol

• • Images with political symbol

The categorization by symbol type, as shown in Figure 6, is based on the type of symbol extracted by GPT and Gemini.

Figure 6. Comparison of social, cultural, and political entities extracted by GPT-4o and Gemini Pro-Vision.

4.5. Adapting SEIZ model for Instagram data

To model the spread of resonance on Instagram using an epidemiological approach, it is crucial to select an appropriate model and parameters that accurately reflect the complexity and realism of the problem. Our methodology draws a parallel between the spread of likes on Instagram and the dissemination of toxicity and rumors on social media (Obadimu et al. Reference Obadimu, Mead, Maleki and Agarwal2020). People’s ideologies are complex, and when they are exposed to news or rumors, they may hold different views, take time to adopt an idea, or even be skeptical of some of the facts. In these situations, they might be persuaded to propagate a story or share it only after careful consideration. Additionally, it is quite conceivable that an individual can be exposed to a story yet never share it themselves.

According to Jin et al. (Reference Jin, Dougherty, Saraf, Cao and Ramakrishnan2013), the SEIZ model has the following rules: Susceptibles, once exposed to a post, transition into the Exposed compartment. Individuals in the Exposed compartment may transition to the Infected class, either after further contact with the Infected or without additional contact through self-adoption, or may become Skeptics. All transitions occur at a specific rate.

Specifically, we identify the following groups of Instagram users for the SEIZ model, which are modeled using equations 1 and demonstrated in Figure 7.

• • $ S(t) $ : Susceptible users who have not seen the narrative content yet but might encounter and engage with it, even if they do not follow infected users.

• • $ E(t) $ : Exposed users who have seen the narrative content and are evaluating whether to share it, typically following infected users.

• • $ I(t) $ : Users actively posting about or amplifying the narrative content.

• • $ Z(t) $ : Users who delay posting after exposure or have stopped actively engaging with the narrative content.

Figure 7. SEIZ model showing the flow between Susceptible (S), Exposed (E), Infected (I), and Zero (Z) states with transition rates.

The following system of Ordinary Differential Equations (ODEs) represents the SEIZ model (Jin et al. Reference Jin, Dougherty, Saraf, Cao and Ramakrishnan2013), demonstrated in Figure 7:

(1) $$ \left\{\begin{array}{ll}\frac{dS}{dt}& =-\frac{\beta SI}{N}-\frac{\gamma SZ}{N},\\ {}\frac{dE}{dt}& =\frac{\left(1-p\right)\beta SI}{N}+\frac{\left(1-\lambda \right)\gamma SZ}{N}-\frac{\eta E I}{N}-\varepsilon E,\\ {}\frac{dI}{dt}& =\frac{p\beta SI}{N}+\frac{\eta E I}{N}+\varepsilon E,\\ {}\frac{dZ}{dt}& =\frac{\lambda \gamma SZ}{N}.\end{array}\right. $$

To apply the SEIZ model from equations (1), key parameters like contact rates ( $ \beta $ , $ \gamma $ , $ \eta $ ) must be defined, along with initial values for $ S\left({t}_0\right) $ , $ E\left({t}_0\right) $ , $ I\left({t}_0\right) $ , and $ Z\left({t}_0\right) $ . The implementation was done in Python using scipy.optimize for least-squares fitting and odeint for solving the ODEs. We used Nelder–Mead optimization within scipy.optimize.minimize to ensure parameter convergence by minimizing the difference between the Infected compartment (I) and the corresponding Instagram data, $ \mid I(t)- posts(t)\mid $ . We tracked weekly cumulative post counts to set optimization boundaries.

6. Policy implications

The findings from this study provide actionable insights for policymakers, platform developers, and researchers working at the intersection of data governance, algorithmic accountability, and digital resilience. Our analysis demonstrates that symbolic content embedded in social media posts, particularly social, cultural, and political symbols, significantly influences user engagement and perceptions of fairness. Cultural symbols, in particular, are associated with increased trust in content, suggesting that visual elements are not only aesthetic features but also critical cues in shaping how users interpret the legitimacy of information.

Given this, content moderation systems should be enhanced to include symbolic awareness. Traditional text-based moderation may miss the deeper narrative cues carried by images. Integrating multimodal language models such as GPT-4o and Gemini Pro Vision allows for better detection of symbolic content and improves the ability of platforms to distinguish between benign cultural expression and potential misinformation. These models can enrich platform moderation capabilities by recognizing symbols that evoke collective identity, political stance, or social sentiment.

The strategic use of symbolic content also has implications for public communication. Campaigns designed to counter misinformation would benefit from incorporating culturally resonant imagery that aligns with the values and experiences of their target audiences. Fact-checking organizations and electoral authorities can improve outreach and influence by producing symbol-rich content that resonates on an emotional and cultural level. This can help build trust and increase the reach of corrective information, especially during elections or in times of social unrest.

In addition, our use of the SEIZ epidemiological model reveals that symbolic content significantly accelerates the spread of narratives on social media. Posts with multiple types of symbolic content exhibited higher infection rates and faster diffusion compared to those without such elements. This insight supports the development of predictive tools that can monitor and respond to emerging misinformation trends based on their symbolic structure. Real-time dashboards informed by narrative contagion modeling would enable policymakers and platform safety teams to act swiftly and strategically in containing harmful content before it escalates.

This research underscores the importance of adopting a symbol-aware approach to digital governance. Understanding the role of visual and cultural symbols in online communication is essential for building trustworthy, inclusive, and resilient digital public spheres. By embedding this awareness into policy and platform design, stakeholders can more effectively safeguard democratic discourse in an age increasingly shaped by algorithmic media and symbolic persuasion.

8. Limitations and future work

While this study provides a comprehensive analysis of symbolic content in social media during Taiwan’s 2024 anti-misinformation campaign, there are several limitations that should be acknowledged. First, the data collection was restricted to Instagram, which, although highly visual and widely used, represents only one platform among many where misinformation and counter-narratives circulate. The findings may not fully generalize to other platforms with different user behaviors, affordances, and content dynamics such as TikTok, X (formerly Twitter), or Facebook.

Second, the classification of symbols relied on the interpretive outputs of large language models, specifically GPT-4o and Gemini Pro Vision. While these models offer advanced multimodal capabilities, they are not free from limitations in cultural sensitivity or contextual accuracy. The symbolic interpretations are influenced by the training data and design of these models, which may introduce biases or misclassifications, especially for more subtle or region-specific symbols.

Third, the study focuses primarily on engagement metrics such as likes and the modeled diffusion of posts, without incorporating deeper behavioral indicators such as sharing, commenting patterns, or sentiment dynamics over time. These aspects could provide a more granular understanding of how symbolic narratives influence discourse and decision-making.

Future research should extend this framework across multiple platforms and cultural contexts to examine whether similar symbolic patterns emerge in different electoral or crisis situations. Comparative studies could explore how the same symbols are received differently across regions or demographics, revealing more about the interaction between symbolism and sociopolitical identity. Additional work could also refine the semiotic detection process by incorporating human-in-the-loop validation or expanding the scope of symbols to include audio and motion, especially on platforms where video dominates user interaction.

Moreover, integrating sentiment trajectory analysis and network diffusion patterns could provide a deeper view into how symbolic posts influence not only engagement but also belief formation and public opinion over time. As generative AI continues to shape content production, future studies should also investigate how synthetic symbolic content compares with authentic imagery in terms of influence and trustworthiness.

Expanding the methodological framework to include qualitative interviews or user surveys may further enrich our understanding of how audiences interpret and respond to symbolic elements. This would help validate model predictions and ground symbolic interpretations in real-world perception. As digital ecosystems evolve, continuing to study how meaning is constructed through symbols remains essential for building accountable and trustworthy communication infrastructures.