Policy Significance Statement
Storytelling is a powerful way to engage stakeholders on social media, yet policy studies have paid little attention to this approach. This research analyzed 85,075 Facebook posts and their 3,402,182 reactions to assess how storytelling elements influence engagement. By leveraging machine learning and natural language processing (NLP) to analyze unstructured social media data, we examined the effectiveness of storytelling in corporate communications. Our findings provide valuable insights for both policymakers and practitioners, helping them better understand how storytelling can enhance stakeholder engagement. By adopting storytelling techniques, policymakers can foster deeper connections, simplify complex messages, and create more compelling narratives that resonate with their audiences, ultimately improving their social media impact.
1. Introduction
While policymakers, such as the European Commission, are taking steps to regulate content on social media to protect citizens from activities that could threaten safety and democracy, their approach tends to center on control and enforcement. For example, The Digital Services Act (DSA) enforces obligations for major platforms to curb illegal content and disinformation (European Commission, n.a). In the actual crowded media environment, it is difficult for various organizations to engage stakeholders in their mission to influence public policy (Bublitz et al., Reference Bublitz, Escalas, Peracchio, Furchheim, Grau, Hamby, Kay, Mulder and Scott2016). Another approach could be to use storytelling to foster engagement and build consensus among stakeholders. As nonprofit organizations, agencies, and local, regional, and national governments navigate the complexities of online stakeholder engagement, the power of storytelling has emerged as a compelling tool to foster connections and meaningful interactions on social media. Stories can captivate the audience’s attention, manifesting as cognitive and emotional engagement (Kreuter et al., Reference Kreuter, Green, Cappella, Slater, Wise, Storey, Clark, O’Keefe, Erwin, Holmes, Hinyard, Houston and Woolley2007). They can trigger emotional reactions (Green, Reference Green2006), and these emotions in turn can play an important role during decision-making (Bourguignon et al., Reference Bourguignon, Boeck and Clarke2020).
Although previous studies have investigated the structuration of stories, less research has studied how narrative content impacts social media stakeholder engagement. Over and above, policy studies have not given enough attention to the effectiveness of storytelling compared to research done in other domains (Jones and McBeth, Reference Jones and McBeth2010). Consequently, our paper’s main research question is: What impact do storytelling elements have on social media stakeholder engagement?
For our study, we gained access to 85,075 Facebook publications from 439 Canadian companies and 3,402,182 reactions (2,625,292 likes, 386,279 shares, and 390,611 comments). This dataset can serve as a proxy for policymakers’ publications, as it would be difficult to obtain such a large volume of posts and reactions directly from that community. Specifically, we evaluate the impact of integrating specific storytelling components into firms’ social media publications on engagement. Those components are plots, characters, and sequences of events or actions arranged in chronological order (Shen et al., Reference Shen, Sheer and Li2015), which have consequences (Gilliam and Flaherty, Reference Gilliam and Flaherty2015). To achieve our objective, Narrative Transportation Theory (NTT) offers a theoretical foundation for understanding how stakeholders engage with and begin to cognitively process information when presented with a story (Huang et al., Reference Huang, Ha and Sun-Hwa2018). NTT explains how audiences become fully involved in the story and develop an emotional connection with the characters (Green and Brock, Reference Green and Brock2000).
Our study indicates that character, sequence of events, and setting—both as individual storytelling components and combined—positively influence social media stakeholder engagement. The next section reviews the literature on storytelling and social media, followed by the presentation of our hypotheses. We then describe the advanced methodology that led to our results. Finally, the discussion highlights how storytelling on social media can foster citizen engagement on topics such as policymaking.
2. Literature review
2.1. Storytelling and policymaking
Academics have recently studied how important storytelling is in influencing policymaking, promoting policy changes, and contributing to sensemaking (Davidson, Reference Davidson2017). For example, in workshops on local energy policy, Mourik et al. (Reference Mourik, Sonetti and Robison2021) found that storytelling can assist in learning, inclusion, and participation while helping to deal with conflicts. These authors also found that stories help people understand others’ perspectives while arousing empathy. Lowndes (Reference Lowndes, Stoker and Evans2016) believes that narratives are important in policymaking because they frame problems, and they help justify policy adoption. Buschke et al. (Reference Buschke, Estreguil, Mancini, Mathieux, Eva, Battistella and Peedell2023) used digital storytelling through the Africa Knowledge Platform (AKP) to enhance engagement with policymakers by presenting 17 interactive narratives. These narratives incorporated visuals, data, and maps to facilitate comprehension across different scientific disciplines. They demonstrated that structuring information as a story improves accessibility and helps bridge the gap between science and policy. Their findings suggest that digital storytelling can increase policymakers’ willingness to interact with scientific knowledge, highlighting its potential as an effective tool for knowledge dissemination and stakeholder engagement. Similarly, Boscarino (Reference Boscarino2022) used an experimental survey to compare how textual versus visual policy narratives influence public opinion regarding the Dakota Access Pipeline (DAPL). The findings demonstrated that while visual narratives significantly enhanced perceptions of issue importance, they did not lead to greater recall of information, nor did they significantly outperform textual narratives in shifting attitudes against the pipeline. Additionally, exposure to visual narratives did not increase participants’ willingness to engage in activism related to the issue. Overall, Boscarino (Reference Boscarino2022) highlighted the nuanced impact of visuals in policy narratives, suggesting that visuals effectively elevate perceived importance but may have limited influence on attitudes and behavioral intentions compared to text-only narratives. McBeth et al. (Reference McBeth, Shanahan, Arrandale Anderson and Rose2013) advanced that the goal of narratives is to steer public opinion in favor of a certain policy outcome. Stories could even mobilize stakeholders around a specific position and encourage them to act (McBeth et al., Reference McBeth, Shanahan, Arrandale Anderson and Rose2013; Davidson, Reference Davidson2017). Policy stakeholders embrace a marketing approach to rally support, even with the risk that narratives can carry misinformation (McBeth et al., Reference McBeth, Shanahan, Arrandale Anderson and Rose2013). In another study, to examine whether policy narratives influence public opinion, Shanahan et al. (Reference Shanahan, Mcbeth and Hathaway2011) demonstrated that policy narratives in the media significantly shape public attitudes in two distinct ways. First, they reinforce existing beliefs among readers who already agree with the narrative’s stance—this is described as “preaching to the choir.” Second, they can change the views of readers who initially hold opposing beliefs—this is referred to as “conversion.” While their study was based on a quasi-experimental design using traditional media articles and a student sample, it highlights the persuasive power of narrative elements in shaping opinion. Lowndes (Reference Lowndes, Stoker and Evans2016) is convinced that policymakers can use storytelling to promote social and legislative changes targeted to their audiences, while social impact organizations should consider stories as a powerful instrument to generate empathy toward difficult issues. Storytelling makes it possible for audiences to recall and process information more effectively (Ramm et al., Reference Ramm, Kopf, Dinter and Hönigsberg2021).
Prior research shows storytelling facilitates sensemaking (Davidson, Reference Davidson2017), promotes inclusion and empathy (Mourik et al., Reference Mourik, Sonetti and Robison2021), and can bridge science and policy (Buschke et al., Reference Buschke, Estreguil, Mancini, Mathieux, Eva, Battistella and Peedell2023). It also plays a role in mobilizing support (McBeth et al., Reference McBeth, Shanahan, Arrandale Anderson and Rose2013), though scholars note the risks of strategic narrative manipulation (Lowndes, Reference Lowndes, Stoker and Evans2016; Davidson, Reference Davidson2017).
However, stories also have a dark side in policymaking. Lowndes (Reference Lowndes, Stoker and Evans2016) points out the ethical problems of who owns the story or the power issues raised by the capability of certain individuals or organizations to master storytelling better than others. Also, policymakers can take the wrong decision after being influenced by a particular narrative which focuses only on a minor aspect of reality (Davidson, Reference Davidson2017). Other times, stories can turn against their promoters, which reinforces the importance of approaching narratives strategically (Davidson, Reference Davidson2017). Narratives are not neutral, as they inherently convey values. These insights support the view that storytelling is both a powerful and ethically complex tool in policy discourse.
2.2. Policy research on social media and on social media stakeholder engagement
In the context of the evolving digital landscape, social media redefine citizens interactions, encourage e-participation to influence policymakers’ decisions (Simonofski et al., Reference Simonofski, Fink and Burnay2021), but represent “a key challenge” for policy academics (McBeth et al., Reference McBeth, Shanahan, Arrandale Anderson and Rose2013). Social media have become an important source of information for policymakers because stakeholders share openly their opinions on these platforms, which in turn should motivate policymakers to react to these publications (Belkahla Driss et al., Reference Belkahla Driss, Mellouli and Trabelsi2019). Social media can be a useful tool for policy development, notably because they are easy to use and are readily accessible and popular (Porwol et al., Reference Porwol, Ojo and Breslin2018).
Studies of social media have mostly focused on behavioral engagement and a quantitative measurement based on the number of likes, comments, and shares (Trunfio and Rossi, Reference Trunfio and Rossi2021). For example, in a study of social media engagement strategies in environmental policy, Liang et al. (Reference Liang, Jia and Meng2024) measured engagement by compiling the number of likes, shares and comments on 1,022 posts from nine social media accounts on X. Others have chosen another measure of engagement like McBeth et al. (Reference McBeth, Shanahan, Arrandale Anderson and Rose2013) who measured the number of viewings of 88 videos on YouTube of an American national park aimed at the protection of bisons, even though, video views “are not appropriate proxies for measures of influence” (p. 177). It is important to measure social media engagement as it can lead to citizen engagement and participation (Simonofski et al., Reference Simonofski, Fink and Burnay2021).
3. Research model and hypotheses
We build on narrative transportation theory (NTT) to propose our four hypotheses about the engagement of stakeholders for firm publication on social media. Narrative transportation is defined as “the process of temporarily leaving one’s reality behind and emerging from the experience somehow different from the person one was before entering the milieu of the narrative” (Green et al., Reference Green, Brock and Kaufman2004, p. 315). NTT provides a theoretical framework to understand how storytelling influences engagement on social media. In a systematic literature review on narrative transportation, Thomas and Grigsby (Reference Thomas and Grigsby2024) found that among the 14 major theories they identified, NTT was the most frequently employed. This highlights the prominence of NTT in research, underscoring its widespread application in studies of narrative engagement. When transported into a story, a person can experience emotions and change attitudes or beliefs (Anaza et al., Reference Anaza, Kemp, Briggs and Borders2020), which can improve narrative engagement defined as “the experience of engaging with a narrative” (Busselle and Bilandzic, Reference Busselle and Bilandzic2009, p. 322).
Narrative transportation elements include identifiable characters, which refer to the personas that the story receiver identifies with, verisimilitude, which refers to the believability of the story, and the imaginable plot, which refers to the temporal sequence of events (time and date) happening in the story (van Laer et al., Reference van Laer, de Ruyter, Visconti and Wetzels2014; Ringler et al., Reference Ringler, Sirianni, Peck and Gustafsson2023). Although “it is hardly within our reach to provide a unanimous definition of a narrative that would suit the needs of all the disciplines” (Sibierska, Reference Sibierska2017, p. 49), most narratologists and linguists agree that narratives include (1) one or many characters, (2) a sequence of events, and (3) a setting. Thus, adopting a linguistic perspective is essential for accurately interpreting narrative and storytelling.
We interpret identifiable characters as named entities, including both human personas and inanimate objects such as products. Characters play a crucial role in narrative comprehension, as research indicates that audiences monitor and switch focus between multiple characters’ goals and motivations (Lin et al., Reference Lin, Dale, McDonald, Collier and Jones2019). In addition, it can impact engagement by facilitating narrative transportation, a state in which story receivers empathize with characters and activate their imagination, leading to a sense of detachment from reality (Green and Brock, Reference Green and Brock2002; Slater and Rouner, Reference Slater and Rouner2002; van Laer et al., Reference van Laer, de Ruyter, Visconti and Wetzels2014). Therefore, we propose the following hypothesis:
H 1 : The presence of a character in a social media publication increases stakeholder engagement.
According to van Laer et al. (Reference van Laer, de Ruyter, Visconti and Wetzels2014), a story typically unfolds through a temporal sequence of events that transition from an initial state to an outcome, guided by the plot, which dictates the chronological flow and thematic framing of these events (Bennett and Royle, Reference Bennett and Royle2004). Thus, we propose the following hypothesis:
H 2 : The presence of a sequence of events in a social media publication increases stakeholder engagement.
Moreover, a story’s setting is essential for providing context, and a story’s location. In particular, it plays an important role in forming and structuring the narrative and framing other story elements (Houghton, Reference Houghton2023). This function is also observed in policy research, where Shanahan et al. (Reference Shanahan, Mcbeth and Hathaway2011) demonstrate that the setting—such as Yellowstone National Park—grounds the policy conflict and enables the framing of characters and policy outcomes in competing media narratives. Such findings suggest that narrative settings may similarly shape audience interpretation and response in other contexts, as posited in the following hypothesis:
H 3 : The presence of a setting in a social media publication increases stakeholder engagement.
Our study aims to provide comprehensive knowledge of how individual story elements, such as characters, sequence of events, and setting, contribute distinctly to stakeholder engagement levels in the B2B social media context. Additionally, we investigate the combined effect of these elements within the story. We therefore propose that:
H 4 : The presence of any element of a story in a social media publication increases stakeholder engagement.
Building upon the theoretical background and prior research findings explained above, our research model is presented in Figure 1.
Figure 1. Research model.
4. Methodology
We evaluate our hypotheses by using Natural Language Processing (NLP). Social media generates a very large quantity of written text that can present challenges to analyze and exploit for policymakers (Simonofski et al., Reference Simonofski, Fink and Burnay2021). Contrary to structured data, which often presents itself in numerical forms with unique meanings, unstructured data like text can present multiple facets, such as the length of words, their rational or emotional messages, or the targeted audience (Balducci and Marinova, Reference Balducci and Marinova2018).
4.1. Natural language processing
“Natural Language Processing (NLP) is the umbrella term that describes an Artificial Intelligence (AI) system’s ability to understand and identify the meaning of human language” (Paschen et al., Reference Paschen, Kietzmann and Kietzmann2019, p. 1415). Many marketing studies utilize unstructured data to predict or classify, which can be effectively accomplished using NLP models (Shankar and Parsana, Reference Shankar and Parsana2022). NLP is also relevant in policy studies as demonstrated by at least one study that analyzed and classified 638 messages on Facebook to provide policymakers with insights into citizens’ discussions on various government services and issues (Belkahla Driss et al., Reference Belkahla Driss, Mellouli and Trabelsi2019). NLP encompasses broad activities such as understanding language syntax—understanding language semantics—which involves assigning meaning to characters, words, or sentences through methods such as Named Entity Recognition (NER), and applied language communication, which refers to tasks like automated summarization of texts, speech recognition, and dialogue creation (Shankar and Parsana, Reference Shankar and Parsana2022).
NER is a technique used in NLP to identify significant terms such as locations, individual names, or dates in a given text (Konkol et al., Reference Konkol, Brychcín and Konopík2015). Today, most academic research on NER in business data analysis includes core subjects like brand name, series name, type name, and product name (Round and Roper, Reference Round and Roper2015).
4.2. Data
This study used 85,075 Facebook publications since the social media platform offers the largest reach among English-language platforms (Golovko and Schumann, Reference Golovko and Schumann2019) and has been used by previous research on policymaking (Lubicz-Zaorski et al., Reference Lubicz-Zaorski, Newlands and Petray2024). Our sample contains posts from 439 Canadian companies listed in the international directory of manufacturing companies on the Dun & Bradstreet website (Table 1).
Table 1. Descriptive statistics of the data
We applied data cleansing to the dataset by removing unwanted tokens, including URLs, hashtags, emojis, and email addresses, because they created noise that our machine learning approach could not interpret, thereby biasing the results. In fact, cleaning the data ensured that the textual content fed into our models more accurately represented the semantic and linguistic features of the posts. We also removed all texts written in languages other than English and used only the publications that contained more than three words to ensure that the publications contained sufficient content for analysis. In fact, this step was critical for improving the quality of the inputs and ensuring that our engagement analysis was based on meaningful language patterns rather than extraneous symbols or metadata.
4.3. Dependent variables (DVs): engagement
Academic research has operationalized social media engagement behavior in various ways, such as combining the number of reactions and shares, reactions and comments, or all three—likes, shares, and comments (Bourguignon et al., Reference Bourguignon, Terho and Hajjem2025). Some studies go further by incorporating additional metrics like video views, impressions, and clicks. In this study, we opted to combine reactions, shares, and comments. To calculate engagement for each publication, we further examined different methods, and to assess the reliability and validity of our engagement scoring methods, we conducted several robustness checks. For example, we compared results obtained by dividing each post’s engagement score by the number of followers of the corresponding company Facebook page versus using raw engagement scores without accounting for follower count. This allowed us to examine whether audience size disproportionately influenced perceived engagement. Additionally, we tested our scoring models with and without the inclusion of outliers, ensuring that extreme values did not unduly skew our findings. We also examined the linguistic composition of posts, analyzing results across different thresholds of meaningful English content—specifically posts with at least 75% and 65% meaningful English—to control for the effects of non-linguistic content on the results. Based on the results of these robustness checks, we considered three methods for constructing the engagement variable: An (1) Unweighted Scoring and two Weighted Scoring, (2) Proportional and (3) Inverse-Probability.
4.3.1. Unweighted scoring
In this method, we simply summed the reactions (likes, comments, and shares) to the publication, obtaining an engagement score
$ {\boldsymbol{s}}_{\boldsymbol{i}} $
(Equation 1) per publication
$ {p}_i $
.
$$ {s}_i={likes}_i+{comments}_i+{shares}_i $$
Next, we normalized all the scores from the company publications by the maximum score, obtaining a list of continuous scores in the range [0,1]. Such scaling is particularly useful when different data features have different units and scales, as it brings them to a common scale without distorting differences in the ranges of values.
4.3.2. Weighted scoring
Since different types of engagement—such as likes, comments, and shares—may carry different levels of significance in reflecting stakeholder interaction and impact, we explored a weighted scoring approach. This method allowed us to more accurately assess and rank posts based on how meaningfully they engaged stakeholders. Weighted scoring was particularly important for ensuring a fair comparison across posts, as it accounts for the fact that not all types of engagement contribute equally to overall stakeholder interaction. Therefore, we selected two weighting methods, the Proportional and Inverse-Probability methods, because they offer complementary ways of handling engagement data.
The Proportional method normalized engagement scores by comparing each post’s interaction to the total interactions in the dataset, ensuring posts with lower overall engagement could still rank highly if they performed well relative to others. The Proportional weighting method is based on the formula shown in Equation (2), which compares the post’s engagement with the total engagement across all posts, ensuring a fair comparison.
$$ Engagement\ Score=\frac{Likes\ for\ Post}{Total\ Likes} + \frac{Shares\ for\ Post}{Total\ Shares} + \frac{Comments\ for\ Post}{Total\ Comments\;} $$
Alternatively, the Inverse-Probability method adjusted for biases by giving higher weights to less frequent actions, such as shares or comments, making them more impactful in the scoring. In other words, the Inverse-Probability method assigned greater weight to less common types of engagement because these reactions often signal stronger stakeholder interest or involvement. Unlike likes, which are quick and low-effort, stakeholders are more likely to comment or share a post when they find it especially relevant or meaningful. Therefore, by giving more influence to these less frequent actions, this method aimed to provide a more balanced and insightful measure of true engagement. Equations (3)–(7) show the formula for the calculation of the Inverse-Probability weighting method, where the inverse of each engagement type’s proportion in the total reactions increases the weight of rarer actions, thereby giving more importance to posts with shares or comments relative to those with just likes.
$$ Total\ Reactions\ Total\ Likes+ Total\ Shares+ Total\ Comments $$
$$ Inverse\ Probability\ of\ Likes=\frac{Total\ Reactions\;}{Total\ Likes\;} $$
$$ Inverse\ Probability\ of\ Shares=\frac{Total\ Reactions\;}{Total\ Shares} $$
$$ Inverse\ Probability\ of\ Comments=\frac{Total\ Reactions\;}{Total\ Comments\;} $$
$$ Score\ of\ Post={\displaystyle \begin{array}{l}\left( Likes\ for\ Post\times Inverse\ Probability\ of\ Likes\right)\\ {}+\hskip2px \left( Shares\ for\ Post\times Inverse\ Probability\ of\ Shares\right)+\Big( Comments\ for\ Post\\ {}\times Inverse\ Probability\ of\ Comments\Big)\end{array}} $$
This combination of methods ensures that all forms of engagement were fairly considered, and the final score truly reflects a post’s relative performance.
4.4. Machine learning
Since this study is the first to use a large amount of data to evaluate the effect of specific named entities (as the elements of the story) on stakeholder engagement, the literature did not contain any annotated data that we could use in our study. We therefore needed to collect and annotate the data ourselves, which was impossible to achieve without using automatic approaches. Thus, we followed an approach based on machine learning, one of the most state-of-the-art methods for data classification and analysis, which had been successfully used before to predict if B2B companies’ Facebook publications would receive high engagement (Saravani et al., Reference Saravani, Boeck and Bourguignon2024). In more detail, we used the spaCy library in Python in this research to automatically detect the named entities in the Facebook publications, which were considered elements of the story.
4.5. Independent variables (IV): named entities
After evaluating several NLP libraries such as NLTK and spaCy for language data preparation and NER, we decided to proceed with spaCy. This decision was based on the availability of all necessary named entity labels within this library, whereas the other tools required programming custom filters. Therefore, spaCy proved to be a more reliable, efficient, and easily reproducible tool. Using this Python library, we detected the nine named entities that contribute to storytelling style in the publications and constructed four formative constructs: SEQUENCE (sequence of events), CHARACTER, SETTING, and the higher-level construct of STORY that encompasses the three first constructs—Our Facebook posts did not necessarily contain all storytelling elements; rather, we evaluated the effect of each element or collectively as a whole, rather than a complete narrative. Equations (8) to (11) display the formulas used to build the named entity constructs:
$$ \mathrm{SEQUENCE}\hskip0.5em =\hskip0.5em \mathrm{TIME}\hskip0.5em \vee \hskip0.5em \mathrm{DATE}\hskip0.5em \vee \hskip0.5em \mathrm{EVENT} $$
$$ \mathrm{CHARACTER}\hskip0.5em =\hskip0.5em \mathrm{PERSON}\hskip0.5em \vee \hskip0.5em \mathrm{PRODUCT} $$
$$ \mathrm{SETTING}\hskip0.5em =\hskip0.5em \mathrm{LOCATION}\hskip0.5em \vee \hskip0.5em \mathrm{GPE}\hskip0.5em \vee \hskip0.5em \mathrm{FACILITY}\hskip0.5em \vee \hskip0.5em \mathrm{ORGANIZATION} $$
$$ {\displaystyle \begin{array}{l}\begin{array}{l}\mathrm{STORY}\hskip0.5em =\hskip0.5em \left(\mathrm{TIME}\hskip0.5em \vee \hskip0.5em \mathrm{DATE}\hskip0.5em \vee \hskip0.5em \mathrm{EVENT}\right)\\ {}\begin{array}{c}\hskip16.119996em \vee \end{array}\end{array}\\ {}\hskip8.119999em \left(\mathrm{PERSON}\hskip0.5em \vee \hskip0.5em \mathrm{PRODUCT}\right)\\ {}\begin{array}{c}\hskip16.119996em \vee \end{array}\\ {}\hskip8.119999em \left(\mathrm{LOCATION}\hskip0.5em \vee \hskip0.5em \mathrm{GPE}\hskip0.5em \vee \hskip0.3em \mathrm{FACILITY}\hskip0.5em \vee \hskip0.5em \mathrm{ORGANIZATION}\right)\end{array}} $$
To operationalize the constructs, we treated them as ordinal discrete variables by counting the frequency of each label present in the publications. However, for STORY, we operationalized it by only considering the existence or non-existence of at least one of the named entities in each of the three constructs, thereby assigning a value of 1–3 to it. STORY could not take the value of 0, as that would imply that there is no independent variable to include in the data subset. For example, if the frequencies were SEQUENCE/DATE = 2, SEQUENCE/TIME = 1, CHARACTER/PERSON = 1, CHARACTER/PRODUCT = 1, and SETTING/GPE = 1, the values of the constructs would be as shown in Table 2.
Table 2. Example values of the constructs
4.6. Evaluation of spaCy
A linguistics Ph.D. student manually coded 370 publications to evaluate spaCy’s performance at programmatically identifying a sample of four named entity test variables. Subsequently, spaCy independently labeled the same sample, and its F1-score was calculated using the linguist’s labels as the golden truth and spaCy’s classifications as predictions. In machine learning, the term “Golden Truth” (also known as “Ground Truth”) describes the most accurate, reliable, and valid data available that serves as a benchmark for comparing and validating a machine learning algorithm’s output. While we could have used Holsti’s intercoder reliability to evaluate the data, we selected the F1-score for evaluation because this measure is more commonly used in machine learning, whereas intercoder reliability is typically employed when two humans code data. F1-score is a trade-off between precision and recall, which measure the model’s accuracy in making positive predictions and its ability to identify all actual positives, respectively. SpaCy achieved an overall F1-score of 80.7% in detecting named entities in the data, indicating good performance given the challenging nature of Facebook publications for NER tools and based on our experience in this study. Table 3 indicates the specific F-1 scores for each of the four named entity test variables.
Table 3. spaCy’s F1-score in NER
5. Results
From the statistics provided in Table 4, it can be inferred that there were outliers in the data—The presence of outliers in the final analysis was to preserve the significant values they carried. In fact, we aimed to identify the factors that contributed to the success of these outliers. The results were, however, similar regardless of the presence or absence of outliers. The statistics of the reactions are shown in Table 5.
Table 4. The publications’ scores
Table 5. Average publication reactions
In total, our analysis of 85,075 Facebook publications by large Canadian manufacturing companies indicated the role of specific named entities in stakeholder engagement and provided several insights into the impact of storytelling on social media engagement. We used spaCy to perform NER and Python to calculate engagement metrics (likes, comments, and shares) to measure the success of publications. Then we categorized named entities into SEQUENCE, CHARACTER, SETTING, and STORY.
Figure 2 shows the scatter and fitted line of the data in the higher-level construct STORY, representing the effect of story on engagement.
Figure 2. The positive correlation between story and stakeholder engagement.
By analyzing the form of the distribution of the data, we observed that it does not follow a Gaussian Distribution (Skewness: 1.414767, Kurtosis: 1.408630). Therefore, we obtained the Spearman Rank Correlation Coefficient from the SciPy library in Python and used it to measure the strength and direction of the relationship between the named entities and the engagement, as Spearman’s rho is suitable for ordinal data that do not follow a Gaussian distribution.
Table 6 shows the detailed results of the Spearman correlation, indicating the presence of a clear statistical significance (based on the very low p-value) between the DVs and IVs even though, the significance is weak based on the ρ (rho).
Table 6. Impact of the ordinal named entity values on engagement
a p-value <0.05 (statistically significant).
The study’s findings show statistically significant relationships between the presence of named entities and engagement; however, the strength of these relationships is weak. As indicated in Table 6, there is a positive correlation between CHARACTER (ρ = 0.112, p < 0.000) and engagement, particularly between the PERSON (ρ = 0.118, p < 0.000) entities and engagement, thus supporting H1 .
A positive correlation between SEQUENCE and engagement (ρ = 0.096, p < 0.000) was also found to be significant, indicating that publications mentioning time, date, or events are slightly associated with increased engagement and thereby supporting H2. Specifically, date entities had the highest correlation (ρ = 0.096, p < 0.000), suggesting that including dates in publications can help engage customers.
Findings indicated a positive, albeit weak, impact in the SETTING category (ρ = 0.135, p < 0.000), with GPE (ρ = 0.108, p < 0.000) and ORGANIZATIONS (ρ = 0.101, p < 0.000) as the most influential items. H3 is confirmed.
There was also a significant relationship between the STORY (ρ = 0.149, p < 0.000) and engagement, thereby providing sufficient evidence to support H4. Since this combined correlation value for all three constructs is superior to any individual correlation, it supports the argument that the combination of narrative elements creates a synergy, making the whole (SEQUENCE, CHARACTER, and SETTING together) more impactful than its individual parts.
Additionally, the results of our statistical analysis revealed that using weighted scores produced results that were largely similar to those obtained through a simple summation of reactions. There are several reasons why the weighted scoring may not have yielded different results from the simple summation approach. One possible reason is that the presence of storytelling elements influenced overall engagement, rather than specific types of reactions. In other words, storytelling may have encouraged stakeholders to interact with posts in general, but not in a way that selectively increased more meaningful reactions like shares or comments. As a result, applying different weights to engagement types did not significantly affect the outcomes. Another consideration is the nature of the dataset itself. The sample size or the distribution of engagement types may not have been varied enough to show a meaningful difference between the two methods. However, in datasets with a wider variety of engagement types, applying weights could help capture more nuanced patterns and yield more informative results.
6. Conclusion
6.1. Discussion
Stories are important for policymakers and for social impact organizations alike. While enhancing stakeholder social media engagement is a desirable goal in policymaking, current research primarily focuses on engagement in other disciplines. The growing appeal of social media presents scholars with emerging opportunities and challenges to fully utilize the potential of storytelling to influence policymaking or for policymakers to promote changes. This research significantly adds to the body of knowledge on storytelling and engagement. It offers insights to policymakers and citizen organizations on how to increase stakeholder engagement through storytelling in social media communication.
The study finds that most elements of storytelling, particularly the story itself, affect engagement, although the effect is generally weak. This might be due to various moderators that can influence the narrative transportation process, such as whether the publication was commercial or non-commercial (van Laer et al., Reference van Laer, Feiereisen and Visconti2019). For example, in the non-commercial domain, personal mentions can increase connection and enhance engagement. Based on the work of van Laer et al. (Reference van Laer, Feiereisen and Visconti2019), which presents digital storytelling as a 3-stage process, the decisions marketers must make at each stage are complex and varied, and these decisions significantly influence the effectiveness of their storytelling efforts. When marketers develop a story, they need to carefully balance all the elements of the narrative to ensure its effectiveness.
During storytelling, stakeholders’ perception of manipulative intent in a story can negatively affect the story’s persuasiveness. For example, a product can play a dual role in content. As a character, it can contribute to the believability of the story and increase narrative transportation; however, it can also introduce a commercial intent that can lead to skepticism. When considering the story receiving stage, the characteristics of the stakeholder, such as their attention, education level, familiarity with the content, and their ability to immerse in a story, also affect the effectiveness of storytelling.
Moreover, the complexity of weighing each engagement type did not add enough value in terms of improving the predictive power or insight we gained from the data. In terms of the nature of our task, this may indicate that all types of engagement might contribute to the outcome in a similar manner, and thus, a simple summation of reactions (likes, shares, and comments) could provide a sufficient representation of post engagement without the need for weight adjustments. In other words, the results suggest that the storytelling elements of the posts play a comparable role in driving stakeholder engagement, regardless of their type and frequency. As such, treating each type of reaction rather than a weighted approach may be sufficient for capturing the impact of storytelling on engagement.
6.2. Implications
Our research critically contributes to the literature by presenting empirical evidence on the role of storytelling in enhancing social media stakeholder engagement in a B2B context. We believe that our approach could be applied to the policymaking context, an area where academics recognize the importance of storytelling in shaping policy decisions, driving policy changes, and aiding in sensemaking, and even uniting stakeholders around a specific stance and motivating them to take action (McBeth et al., Reference McBeth, Shanahan, Arrandale Anderson and Rose2013; Davidson, Reference Davidson2017).
First, although the literature is growing, scholars have not yet focused on how narrative content affects stakeholder engagement on social media or how they react to narratives. Our research addresses this gap by using advanced NLP tools to analyze a large quantity of unstructured data. We agree with Belkahla Driss et al. (Reference Belkahla Driss, Mellouli and Trabelsi2019) that NLP is relevant in policy studies to analyze and classify large amounts of data. This method offers a new way to assess the effectiveness of social media content and could be applied to social media content created by policymakers.
Second, our study underscores the importance of specific storytelling elements such as character, sequence of events, and setting in successful social media narratives. It contributes to a more comprehensive theoretical understanding of how narrative structure influences stakeholder engagement. Furthermore, the study identifies specific linguistic patterns associated with higher engagement in social media narrative content.
Thirdly, findings from this study can help enhance policymakers’ strategies, specifically in creating effective narrative content for social media stakeholder engagement. Our research can guide policymakers in using these criteria to evaluate and enhance their existing content strategies and to tailor their narrative content to better align with the preferences and interests of their target stakeholders.
6.3. Limitations and future research
We analyzed 85,075 Facebook publications from Canadian firms to establish if story components had a positive effect on engagement. We encourage other researchers to apply our methodological approach to analyze more specifically social media posts on public policy to determine whether storytelling is favorable to citizen engagement. Our research provides an empirical foundation for future research to explore the nuanced relationship between language use and engagement in policy studies.
We relied on SpaCy for named entity recognition. Although SpaCy is well-suited for high-accuracy tasks and performance in NER tasks, Python libraries are usually designed for standard texts. Therefore, for named entity recognition on data like Facebook publications, a future opportunity would be to train a model specifically with Facebook publications.
Engagement metrics can be skewed by Facebook’s algorithm, promoted publications (a form of paid advertisement), automated bots, and paid likes. However, we considered their effects diluted due to our use of a very large range of publications, which introduces a new approach to evaluate the effectiveness of social media content. We obtained public engagement metrics on Facebook, which are likes, comments, and shares. Future research with access to more complete dashboards could examine the effect of narratives on click-through rates (CTR) and even conversions.
Our model covered only English texts because we used an NER designed specifically for this language. Future studies could explore other libraries or combinations of tools to adapt to other languages and provide additional insights. Future research could conduct comparative analyses with alternative libraries such as NLTK, Stanford NER, or Allen NLP, revealing differences in entity recognition capabilities and improving the results. Moreover, qualitative research, such as in-depth interviews with stakeholders, can support findings and gain a deeper understanding of stakeholders’ emotions, identification, and reflections regarding the stories.
Our study emphasizes that employing content containing references to some story elements is a practical strategy to craft social media behavioral engagement. However, it was limited to certain word categories and excluded images.
Further research is needed to explore the effectiveness of visual elements in stakeholder engagement to determine the best practices for integrating multimodal content in policy social media strategies. Additionally, using machine learning to perform policy research is rich with opportunity. Future studies can employ Multimodal Named Entity Recognition (MNER) to identify entities and classify them with the help of an associated image.
Data availability statement
[dataset] Boeck, Harold; Bourguignon, Benoit; Hosseini Saravani, Seyed Habib; Bahra, Maryam, 2025, “Replication data for: Evaluating the Impact of Storytelling Elements on Social Media Engagement: An AI-driven approach,” https://doi.org/10.5683/SP3/RC5X0E, Borealis, V1, UNF:6:Lk1g + FdlCKEFD/n44vRhiA== [fileUNF].
Author contributions
Conceptualization: Harold Boeck; Benoit Bourguignon; Seyed Habib Hosseini Saravani; Maryam Bahra Methodology: Harold Boeck; Benoit Bourguignon; Seyed Habib Hosseini Saravani; Maryam Bahra Software: Harold Boeck; Seyed Habib Hosseini Saravani Formal Analysis: Harold Boeck; Seyed Habib Hosseini Saravani Writing original draft: Harold Boeck; Benoit Bourguignon; Seyed Habib Hosseini Saravani; Maryam Bahra Supervision: Harold Boeck; Benoit Bourguignon Funding acquisition: Harold Boeck; Benoit Bourguignon Writing—review & editing: Harold Boeck; Benoit Bourguignon; Seyed Habib Hosseini Saravani; Maryam Bahra All authors approved the final submitted draft.
Funding statement
This study draws on research supported by the Social Sciences and Humanities Research Council of Canada.
Competing interest
The authors declare none.
Open access
Comments
No Comments have been published for this article.