3.1 5W1H semantic segmentation

In the initial stage, we segment the tweets by labelling them with the 5W1H components: who, what, when, where, why, and how. The 5W1H components are considered essential for covering a piece of information and therefore form the basis for information extraction. In journalism, a report is considered complete only if it answers the question of Who did what to whom, when, where, why, and how. For example, the given sentence “John met Sara privately, in the hall, on Monday to discuss their relationship” describes the event “someone meeting someone.” We, therefore, model the tweets as a sequence of 5W1H (Who, What, When, Where, Why, and How) semantic components. Let $w = \left \{ w_1, w_2,{\ldots }, w_n \right \}$ be the sequence of words in a tweet and $A$ be the attribute to which $w$ is to be mapped. Therefore, a tweet with the 5W1H components may be considered as $\langle w, A \rangle$ , where, $A$ is the tuple $\langle WHO, WHAT, WHEN, WHERE, WHY, HOW \rangle$ in 5W1H. Let $w$ = “John met Sara privately, in the hall, on Monday to discuss their relationship.” In this example sentence, the 5W1H components will be identified as:

\begin{gather*} \left \langle John\right \rangle _{\left [\textbf{WHO}\right ] } \left \langle met\;Sara \right \rangle _{\left [ \textbf{WHAT}\right ]} \left \langle privately \right \rangle _{\left [ \textbf{HOW}\right ]}\!, \left \langle in\;the\;hall\right \rangle _{\left [\textbf{WHERE}\right ]}\!, \\ \left \langle on\;Monday\right \rangle _{\left [\textbf{WHEN}\right ]} \left \langle to\; discuss\;their\;relationship\right \rangle _{\left [\textbf{WHY}\right ]} \end{gather*}

The set of words contained in the text $w$ is represented by $\psi _{A}\left ( w \right )$ which is classified as the attribute $A$ where $A \;\in \;5W1H$ . Therefore, the 5W1H model of tweets can be represented as:

(1) \begin{equation} \psi _{5W1H}\left ( w \right ) =\bigcup _{A\;\in \;5W1H} \psi _{A}\left ( w \right ) \end{equation}

Equation (1) suggests that a tweet is a sequence of words where the words are grouped into the 5W1H components. Due to its short length, a tweet may or may not have all of the 5W1H components present. Therefore, in equation (1), the 5W1H model of the tweets is the union ( $\bigcup$ ) of the set $\psi _{A}$ containing the words $w$ where $A$ could be any of the 5W1H components. Therefore, we model 5W1H as a sequence labelling task. Since each 5W1H component is either a single word or a phrase, we adopted the BIO encoding scheme for labelling the tweets, where a “B” indicates the beginning of the 5W1H component, an “I” indicates that a word is inside of a 5W1H component, and an “O” indicates that the word is outside of 5W1H (does not represent any of the 5W1H components). The BIO sequence labelling strategy is shown in Fig. 2.

Figure 2.

5W1H BIO sequence labelling example.

For this purpose, we used a deep neural network system implemented by Chakma, Das, and Debbarma (Reference Chakma, Das and Debbarma2019) that gave an F-1 score of 88.21% with an error of 11.79% on the dataset. Since there is an error of 11.79%, we corrected them by manually relabelling the misclassified tweets. In some cases, the deep learning system did not correctly label the BIO-based tag sequences of the 5W1H components. The following example illustrates the misclassification scenario.

Example Tweet:

\begin{equation*} \textit{the way trump will give his victory speech will define his presidency} \end{equation*}

The above tweet has two main verbs: give and define. This represents two events (“giving” and “defining”). If we consider the “giving” event, then the 5W1H sequence tagging/labelling is done as:

\begin{gather*} \left \langle the\right \rangle _{\textbf{B-How}} \left \langle way\right \rangle _{\textbf{I-How}} \left \langle trump\right \rangle _{\textbf{B-Who}} \left \langle will\right \rangle _{\textbf{O}} \left \langle give\right \rangle _{\textbf{verb}} \left \langle his\right \rangle _{\textbf{B-What}} \left \langle victory\right \rangle _{\textbf{I-What}}\\ \left \langle speech\right \rangle _{\textbf{I-What}} \left \langle will\right \rangle _{\textbf{O}} \left \langle define\right \rangle _{\textbf{O}} \left \langle his\right \rangle _{\textbf{O}} \left \langle presidency\right \rangle _{\textbf{O}} \end{gather*}

Whereas, if we consider the “defining” event, then the 5W1H sequence tagging/labelling is done as;

\begin{gather*} \left \langle the\right \rangle _{\textbf{B-Who}} \left \langle way\right \rangle _{\textbf{I-Who}} \left \langle trump\right \rangle _{\textbf{I-Who}} \left \langle will\right \rangle _{\textbf{I-Who}} \left \langle give\right \rangle _{\textbf{I-Who}} \left \langle his\right \rangle _{\textbf{I-Who}} \left \langle victory\right \rangle _{\textbf{I-Who}}\\ \left \langle speech\right \rangle _{\textbf{I-Who}} \left \langle will\right \rangle _{\textbf{O}} \left \langle define\right \rangle _{\textbf{verb}} \left \langle his\right \rangle _{\textbf{B-What}} \left \langle presidency\right \rangle _{\textbf{I-What}} \end{gather*}

In the case of the “defining” event, the Who component covers “trump” which is labelled as $\left \langle trump\right \rangle _{\textbf{I-Who}}$ . However, the system misclassified it as $\left \langle trump\right \rangle _{\textbf{B-Who}}$ . Therefore, it is necessary to manually correct this type of misclassification. We then convert the 5W1H-labelled tweets into a structural vector representation $e^{5w1h}$ shown in Fig. 3. Then a contextualised vector $e^{contextual}$ is obtained by concatenating $e^{5w1h}$ with embeddings of tweets and embeddings of verbs:

(2) \begin{equation} e^{contextual}=\left [e^{tweet}\circ \;e^{5w1h}\circ \;e^{verb}\right ], \end{equation}

where $e^{tweet}$ is the word embeddings of the tweet, $e^{verb}$ are the verb embeddings, and $e^{5w1h}$ is the 5W1H embeddings. It is not necessary that all 5W1H components are always present in a tweet. For example, in the sentence “Donald Trump won the elections,” there is no “When,” “Where,” “Why,” and “How” components present. Therefore, to generate fixed-sized embeddings, the embeddings are padded with zeros (0’s). The input to BERT is given in the format $\left \langle \left [CLS\right ] tweet \left [SEP\right ] 5W1H\left [SEP\right ]verb \left [SEP\right ] \right \rangle$ (Fig. 3), where the sequence $\left [SEP\right ]5W1H\left [SEP\right ]$ is the Who, What, Where, When, Why, and How encoding of the sentence. A “verb indicator” embedding is then concatenated into the 5W1H encoding to distinguish the verb tokens from the nonverb ones. The sequences ( $[e^{tweet}, e^{5w1h}, e^{verb}]$ ) are then fed into the BERT network to generate contextual embeddings $e^{contextual}$ . Contextual embeddings are obtained by aggregating (concatenating) them on the last axis, resulting in sentence encoding of 1,536 dimensions.

Figure 3.

Contextualised vector generation.

3.2 Event clustering

We group the contextual embeddings generated by BERT into clusters where each cluster corresponds to a possible event (or sub-event). For clustering contextual embeddings, we applied four clustering techniques such as k-means (Jin and Han Reference Jin and Han2017), Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) (Campello, Moulavi, and Sander Reference Campello, Moulavi and Sander2013), Hierarchical Agglomerative Clustering (HAC) (Zepeda-Mendoza and Resendis-Antonio Reference Zepeda-Mendoza and Resendis-Antonio2013), and Affinity Propagation (AP) (Frey and Dueck Reference Frey and Dueck2007). k-means requires the number of clusters to be specified before starting the algorithm. Since arbitrarily selecting $k$ is not a viable approach; we, therefore, selected the number of clusters $k$ based on quality metrics such as Silhouette (Rousseeuw Reference Rousseeuw1987) for the given dataset. Silhouette score Footnote ^c is a method for measuring the similarity of a data object within the same cluster to that of other clusters. The Silhouette score varies between $-1$ and $+1$ . Higher values indicate higher similarity of a data object within its own cluster and poor similarity with neighbouring clusters. The Silhouette score for a sample $s$ is measured as:

(3) \begin{equation} s=\frac{b-a}{max\left (a,b\right )} \end{equation}

where an increase in Silhouette score for $k$ represents the suggested number of clusters. We used the scikit-learnFootnote ^d API for implementing k-means clustering.

HDBSCAN is an extension of the DBSCAN (Ester et al. Reference Ester, Kriegel, Sander and Xu1996) algorithm that converts DBSCAN into a hierarchical clustering algorithm. HDBSCAN views data as a weighted graph with vertices and edges, with weights equal to the mutual reachability distance between the points. Using the weighted graph, HDBSCAN extracts flat clustering based on the stability of the clusters. The algorithm goes through the following steps: (1) Transform the space according to the density/sparsity of the data distribution, (2) Build the minimum spanning tree of the distance weighted graph, (3) Construct a cluster hierarchy of connected components, (4) Condense the cluster hierarchy based on the minimum cluster size, and (5) Extract the stable clusters from the condensed tree. The algorithm considers dense data distribution as “islands” and sparse noise as “sea.” The objective is to find the islands of higher density from a sea of sparser noise. That means making "sea" points more distant from each other and from the “island.” Campello et al. (Reference Campello, Moulavi and Sander2013) defines core distance as the distance of a point to the $k{\rm th}$ nearest neighbour which is denoted as $\mathrm{core}_k\left (x\right )$ for parameter $k$ for a point $x$ . The distance metric between points is measured with mutual reachability distance which is represented as:

(4) \begin{equation} d_{mreach-k}( a,b ) = max\left \{ core_{k}( a ), core_{k}( b ), d( a,b)\right \} \end{equation}

where $d(a,b)$ is the original metric distance between $a$ and $b$ . All the low core distance points remain at the same distance from each other, but the larger core distance points are pushed away to be at least their core distance away from any other point. This process lowers the sea level, thereby spreading sparse sea points out while leaving land untouched. However, it depends on the choice of $k$ where larger $k$ values interpret more points as being in the sea. The algorithm then constructs a minimum spanning tree with the data points as vertices and an edge between any two points with a weight equal to the mutual reachability distance of those points. The minimum spanning tree is then converted into a hierarchy of connected components. This is most simply accomplished by sorting the tree’s edges by distance (in increasing order) and then iterating through, establishing a new merged cluster for each edge. The initial phase of cluster extraction involves the condensation of the intricate and extensive cluster hierarchy into a more concise tree, wherein each node contains a slightly augmented dataset. Upon examination of the aforementioned hierarchy, it is often observed that a cluster split emerges as a result of one or two points branching off from a cluster. It is critical to recognise that this split should be viewed as a persistent, single cluster that undergoes a loss of points, rather than the formation of two new clusters. To solidify this concept, it is essential to establish a minimum cluster size, which serves as a parameter for HDBSCAN. Once this minimum cluster size is determined, we can proceed to traverse the hierarchy and inquire at each split whether one of the newly formed clusters contains fewer points than the minimum cluster size. If this is the case, we declare it as “points falling out of a cluster” and attribute the larger cluster with the parent’s cluster identity, while recording the points that “fell out” and the corresponding distance value. Conversely, if the split leads to two clusters, each at least as significant as the minimum cluster size, we acknowledge it as a genuine cluster split and allow it to persist in the tree. By following this process and traversing the entire hierarchy, we obtain a considerably smaller tree with only a few nodes, each of which contains information regarding the cluster’s size at that particular node and the subsequent decrease in size as the distance varies. We used the pythonFootnote ^e -based implementation of HDBSCAN for our purpose.

HAC comes under the family of hierarchical clustering algorithms that build nested clusters by merging them successively with a bottom-up approach. The merging could be done by either one of the four approaches: ward, complete linkage, average linkage, and single linkage. Ward minimises the sum of squared differences within all clusters. Complete linkage minimises the maximum distance between observations of pairs of clusters. Average linkage minimises the average of the distances between all observations of pairs of clusters. Single linkage minimises the distance between the closest observations of pairs of clusters. We used the scikit-learnFootnote ^f API for implementing HAC clustering with linkage parameters set to ward and complete linkage. For ward, we set the affinity parameter to “euclidean” and for complete linkage to “cosine,” respectively. We did not observe a significant difference in the generated clusters by both methods. In scikit-learn, HAC requires a threshold value to be set for either one of the two parameters: n_clusters and distance_threshold where the former represents the number of clusters to be generated, whereas the latter represents the linkage distance threshold above which clusters will not be merged. Selecting a distance threshold is difficult. Therefore, we specified the number of clusters $k$ based on two evaluation parameters: silhouette score and dendogram cutoff. Both Silhouette score and dendogram cutoff suggested $k=10$ for the given contextual embeddings.

In AP, clusters are created by sending messages between pairs of samples until convergence. A concept called “exemplar” is used which is the most representative of other samples in a given dataset. The messages actually represent the suitability of one sample to be the exemplar of the other. This is then updated based on the values received from the others. This updating process continues iteratively until convergence, and then the final exemplars are chosen thus giving the final cluster. The messages fall into two categories as per scikit-learn API:Footnote ^g (a) the accumulated evidence called responsibility denoted as $r\left (i,k \right )$ such that sample $k$ should be the exemplar for a sample $i$ and (b) availability denoted as $a\left ( i,k\right )$ which is the accumulated evidence that sample $i$ should choose sample $k$ as its exemplar. The basic idea is that exemplars are chosen by samples if they are (1) similar enough to many samples and (2) chosen by many samples to be representative of themselves. The responsibility of a sample to be the exemplar of sample is given by:

(5) \begin{equation} r\left ( i,k \right )\leftarrow s\left ( i,k \right )- max\left [ a\left ( i,{k}' \right )+ s\left ( i,{k}'\right ) \forall{k}'\neq k \right ] \end{equation}

where $s\left (i,k \right )$ is the similarity between samples $i$ and $k$ . The availability of sample $k$ to be the exemplar of sample $i$ is given by:

(6) \begin{equation} a\left ( i,k \right )\leftarrow min\left [ 0,r\left ( k,k \right )+\sum _{{i}^{\prime}s.t.{i}^{\prime}\not{\varepsilon }{i,k}}r\left ({i}^{\prime},k \right ) \right ] \end{equation}

Initially, all values for $r$ and $a$ are set to zero, and the calculation of each iterates until convergence.

3.3 Event summarisation

There are two broad categories of text summarisation: (1) extractive and abstractive. Extractive summarisation is the task of generating a summary of a document by identifying the most important sentences (or phrases) and then concatenating them without the inclusion of any new words. Abstractive summarisation is the task of generating a summary of a document by paraphrasing the sentences (or phrases) and producing newer phrases. For our work, we explore the abstractive summarisation approach. Recently, the natural language processing (NLP) domain has observed significant growth in neural network-based solutions for the text summarisation task. Encoder–decoder (Sutskever, Vinyals, and Le Reference Sutskever, Vinyals and Le2014) networks are generally used for the summarisation task where the encoder could be a Recurrent Neural Network (RNN) (Sherstinsky Reference Sherstinsky2020) or a transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) network and the decoder could be autoregressive or non-autoregressive. In the autoregressive approach, the decoder can make current predictions with knowledge of previous predictions. For the abstractive summarisation task, we explored three neural network architectures:

• • Pointer-generator (Vinyals, Fortunato, and Jaitly Reference Vinyals, Fortunato and Jaitly2015)-based network (See et al. Reference See, Liu and Manning2017) (we denote it as P-Gen)

• • Presumm (Liu and Lapata Reference Liu and Lapata2019), a transformer-based implementation.

• • BART (Lewis et al. Reference Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov and Zettlemoyer2020), a combination of Bidirectional and Autoregressive Transformers.

The above-mentioned three architectures have shown promising results in abstractive summarisation tasks on state-of-the-art datasets (Hermann et al. Reference Hermann, Kocisky, Grefenstette, Espeholt, Kay, Suleyman and Blunsom2015). Therefore, we used the pre-trained models of P-Gen, Presumm, and BART on our dataset and performed several experiments to evaluate the generated summaries.

3.3.1 Pointer-generator (P-Gen)

P-Gen is an encoder–decoder network with Long Short Term Memory (LSTM) at the encoder side and a pointer-generator network at the decoder side. It is a hybrid of the sequence to sequence (Nallapati et al. Reference Nallapati, Zhou, dos Santos, GuÌçlehre and Xiang2016) attention and pointer network. P-Gen also exploits the concept of coverage (Tu et al. Reference Nallapati, Zhou, dos Santos, GuÌçlehre and Xiang2016) mechanism which is added to the pointer-generator network. Here, we reuse the definition of coverage vector and the other parameters defined in P-Gen. Coverage is a vector $c^{t}$ which is the sum of the attention distributions of all previous decoder time steps represented as:

(7) \begin{equation} c^{t}=\sum _{t^{\prime}}^{t-1}a^{t^{\prime}} \end{equation}

where $a^{t^{\prime}}$ is the attention distribution. The coverage vector is basically a modification of the original attention vector of Bahdanau et al. (Reference Bahdanau, Cho and Bengio2015). The modified attention vector is represented as:

(8) \begin{equation} e^{t}_{i} = v^{T}tanh\left ( W_{h}h_{i} + W_{s}S_{t} + w_{c}c^{t}_{i} + b_{attn} \right ) \end{equation}

where $v^{T}$ , $W_{h}h_{i}$ , $W_{s}S_{t}$ , $w_{c}c^{t}_{i}$ , and $b_{attn}$ are learnable parameters. P-Gen also define coverage loss which penalises the decoder for repeatedly attending the same locations. The coverage loss is represented as:

(9) \begin{equation} covloss_{t}=\sum min\left (a^{t}_{i},c^{t}_{i}\right ) \end{equation}

The final composite loss function in P-Gen is defined as:

(10) \begin{equation} loss_{t} = -logP\left ( w^{*}_{t} \right ) + \lambda \sum min\left (a^{t}_{i},c^{t}_{i}\right ) \end{equation}

where $\lambda$ is some hyperparameter.

3.3.2 Presumm

It is an encoder–decoder network with a transformer network on the encoder and decoder side. Presumm presents a two-stage approach where the extractive summarisation is performed in the first stage and then the abstractive summarisation in the second stage. The encoder is therefore fine-tuned twice. Since Presumm is based on the BERT architecture, special tokens [CLS] and [SEP] are inserted between each sentence to generate positional embeddings. Interval segment embeddings are generated to differentiate each sentence. For example, assuming a document $D$ consisting of sentences $\left [ sent_1, sent_2, sent_3, sent_4 \right ]$ , embeddings can be generated as $\left [ E_A, E_B, E_C, E_D \right ]$ . For the extractive summarisation, vector $t_i$ of the $i^{th}$ [CLS] token from the top layer is used to represent $sent_i$ . Several transformer layers are stacked together to obtain document-level features for extracting summaries:

(11) \begin{equation} \tilde{h^l} = LN\left ( h^{l-1} + MHAtt\left ( h^{l-1} \right ) \right ) \end{equation}

(12) \begin{equation} \tilde{h^l} = LN\left ( h^{l} + FFN\left ( h^{l} \right ) \right ) \end{equation}

where $LN$ is the layer normalisation operation (Ba, Kiros, and Hinton Reference Ba, Kiros and Hinton2016), $MHAtt$ is the multi-head attention (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), $h^{0} = PosEmb(T)$ , $T$ denotes the sentence vectors, and function $PosEmb$ adds positional embeddings to $T$ . Sigmoid is used at the final output layer, which is represented as:

(13) \begin{equation} \hat{y_i} = \sigma \left ( W_{o}h_{i}^{L} + b_o \right ) \end{equation}

where $h_{i}^{L}$ is the vector for $sent_i$ from the top layer (the $L^{th}$ layer ) of the transformer. For abstractive summarisation, the presumm, uses the pre-trained encoder of the extractive summariser and a six-layer transformer for the decoder.

3.3.3 BART

In BART, the sequences are encoded as denoising autoencoders. The training data for BART contain “corrupted” or “noisy” text, which will be mapped to the original or clean text. The training format is similar to the denoising autoencoders. Noising schemes include Token Masking, Token Deletion, Text Infilling, Sentence Permutation, and Document Rotation. The first part of BART uses the BERT’s bidirectional encoder to find the best representation of the input sequence. For each sequence of text in the input, the BERT encoder outputs an embedding vector for each token in the sequence, plus an additional vector containing sentence-level information. After obtaining the tokens and sentence-level representations of the input text sequence, the decoder must interpret them to match the output destination. Therefore, causal or autoregressive models that look only at past data to predict the future are practical. The decoder section of conventional transformers and the GPT-1 type shared a similar construction. Only learning from previous tokens can affect the computation of current tokens because GPT stacks twelve of these decoders sequentially. The GPT decoder uses feedforward layers with masked multi-head self-recognition blocks, just like the original transformer decoder. The model is initially pre-trained on tokens t looking back to k tokens in the past to compute the current token. To enable the model to “learn the language,” this is done unsupervised on a sizable text corpus:

(14) \begin{equation} L_{1}\left (T \right )= \sum _{i} logP\left (t_{i}\vert t_{i-k},{\ldots }, t_{i-1};\ \theta \right ) \end{equation}

The model is then fine-tuned in a supervised manner to maximise the likelihood of a label y given feature vectors $x_{1}{\ldots }x_{n}$ :

(15) \begin{equation} L_{2}\left (C\right )=\sum _{x,y}logP\left ( y \vert x_{1},{\ldots }, x_{n}\right ) \end{equation}

Equations (14) and (15) are combined to get the objective in equation (16), where the lambda is a learned weight parameter that limits the impact of language modelling:

(16) \begin{equation} L_{3}\left ( C \right )=L_{2}\left (C\right )+\lambda L_{1}\left (C\right ) \end{equation}

4.1 Dataset

We collected tweets on three different topics using twitter4jFootnote ^h API: Demonetization,Footnote ⁱ US Elections 2016 Footnote ^j , and Me Too India Movement.Footnote ^k For Demonetization, we crawled around 14,940 tweets during two different time periods: one between $22^{\rm nd}$ and $23^{\rm rd}$ November 2016; and the other between $11^{\rm th}$ and $21^{\rm st}$ April 2017. The intention for the first period was to extract tweets related to the Demonetization topic after its announcement by Govt. of India on $8^{\rm th}$ November 2016. The second period was intended to extract tweets that discuss the impact of demonetisation on the new financial year 2017–2018 that began on $1^{\rm st}$ April 2017. For US Elections 2016, we crawled 38,984 tweets on the day of the elections, that is, $8^{\rm th}$ November 2016, and on the final declaration of the electoral results on $17^{\rm th}$ January 2017. The tweets were collected with hashtags such as “#USElections2016,” “#ElectionNight,” “#DonaldTrump,” “#HillaryClinton,” “#DonaldTrumpWins,” and “#USElections2016Results.” Apart from hashtags, we also used terms such as “Donald Trump,” “Trump,” “Hillary Clinton,” “Hillary,” and “Clinton.” For the Me Too topic, we crawled a total of 248,160 tweets with queries such as “#MeToo,” “#MeTooCampaign,” “#MeTooControversy,” and “#MeTooIndia” from $11^{\rm th}$ to $24^{\rm th}$ October 2018.

We performed some preprocessing tasks by lower-casing the tweets, removing Non-English tweets, and retweets. We observed that approximately 80% of the collected tweets contained retweets as well as non-English tweets. Therefore, after preprocessing, we finalised a total of 16,375 tweets: 3,484 on “Demonetization,” 6,558 on “Me Too India Movement,” and 6,333 on “US Elections 2016.” Due to data privacy reasons, the corpus is not available publicly. The dataset will be available on request.

Figure 4.

Cluster selection process for the summarisation task.

4.2 Cluster selection

In this subsection, we discuss the criteria for the selection of clusters for the summarisation task. This procedure is shown in Fig. 4. We feed the contextual embeddings $e^{contextual}$ to four clustering algorithms (k-means, HAC, HDBSCAN, and AP). For k-means and HAC clustering, we choose the number of clusters $k$ based on the suggested Silhouette score. As shown in Fig. 5(a) and (b), we set the number of clusters $k=10$ for both the k-means and HAC clustering algorithms. Therefore, both k-means and HAC generated ten clusters. For k-means, we set the following parameters: n_clusters = best_k, n_jobs = −1, and random_state = seed. The parameter best_k = 10 is chosen based on the Silhouette Score iterations between 2 and 50. The parameter n_jobs refers to the number of parallel threads to use for the computation. A value of “−1” means that all available processors will be used for the computation. The parameter random_state = seed determines a random number generation for centroid initialisation. This is done to produce the same results across different calls. We use the popular seed value of 42 for the generation of random numbers. For HAC, we set the following parameters: affinity = ‘euclidean’ or ‘cosine’, linkage = ‘ward’ or ‘complete_linkage’, and n_clusters = 10. For HDBSCAN, we set the standard parameter settings: min_cluster_size = n,Footnote ^l prediction_data = True, core_dist_n_jobs = −1, and memory = ’data’. For AP clustering, we use the default parameter settings such as damping = 0.5, preference = None, and affinity = ‘euclidean’.

Figure 5.

Selection of best value for number of clusters $k$ for k-means and HAC.

Figure 6.

Visualisation of generated clusters.

Cluster formation is shown in Fig. 6 and the population distribution in Fig. 7. As can be seen in Fig. 6(a) and (b) and Fig. 7(a) and (b), both the k-means and the HAC generated ten clusters, respectively. AP generated a total of 934 clusters (Figs. 6(c) and 7(c)). It produced cluster sizes between 2 and 115 tweets. The average size of the clusters produced by AP is 17.53 tweets. HDBSCAN (Figs. 6(d) and 7(d)) was unable to group 10,057 (i.e., 61.4% of 16,375) tweets and labelled them “−1”. For the remaining 6,318 tweets, HDBSCAN generated 2,356 clusters. It produced a minimum cluster size of two tweets and a maximum of twenty-three tweets, which makes it insignificant for generating summaries. We therefore ignored HDBSCAN generated clusters because 61.4% of the tweets were not considered in any cluster (labelled “−1”). Therefore, excluding HDBSCAN, we measured the cosine similarity of tweets in each cluster ( $c_i$ ) produced by k-means, HAC, and AP approaches. As shown in Fig. 4, we set the minimum threshold of the similarity score to 70%. If more than or equal to 50% of the tweets in a cluster fall below the minimum similarity threshold of 70%, the cluster and the corresponding clustering method (i.e., k-means/HAC/AP) are not considered. Based on observations of cosine similarity scores, only AP clusters with a cluster size of a minimum of ten tweets are considered for the summarisation task. The first few tweets from the dataset labelled by k-means, HAC, HDBSCAN, and AP are shown in Table 1. The table shows the results of applying different clustering approaches to the dataset. For example, k-means and HAC generated ten clusters with labels 0–9. HDBSCAN could not generate clusters for more than 60% of the data and labelled them with −1. HDBSCAN generated clusters only for the remaining 40% of the data with cluster labels between 0 and 2,355. AP generated 934 clusters with labels ranging from 0 to 933.

Table 1.

A snapshot of the cluster of tweets labelled by k-means, HAC, HDBSCAN, and AP

Figure 7.

Population distribution of clusters by k-means, HAC, AP, and HDBSCAN. (a) Ten clusters generated by k-means clustering with the distribution of 1,466, 2,280, 1,073, 1,481, 2,073, 834, 1,615, 1,768, 2,117, and 1,668 tweets. (b) Ten clusters generated by HAC clustering with the distribution of 2,267, 2,170, 3,004, 594, 3,928, 1,868, 100, 866, 385, and 293 tweets. (c) A total 934 clusters were generated by AP with minimum and maximum cluster size of 2 and 115 tweets, respectively. (d) HDBSCAN failed to cluster 10,057 tweets which is 61.4% of the dataset. The remaining 38.6% are clustered into 2,356 clusters with length between 2 and 23 tweets.

4.3 Reference summary creation

As there is no available dataset for abstractive summarisation on tweets, we had to build our own reference summaries (gold summaries). Since the number of clusters produced by AP is large (a total of 934 clusters out of which we considered only those having a minimum of 10 tweets per cluster) as shown in Fig. 7, manually evaluating the clusters for generating reference summaries is a laborious task. Therefore, we adopted the approach shown in Fig. 8 to generate reference summaries. In the first step, we applied four extractive summarisation techniques such as Lexrank (Erkan and Radev Reference Erkan and Radev2004), LSA (Gong and Liu Reference Gong and Liu2001), Luhn (Luhn Reference Luhn1958), and Textrank (Mihalcea and Tarau Reference Mihalcea and Tarau2004) to obtain four candidate sets ( $s_1, s_2, s_3, and\;s_4$ ). Then in the second step, we asked two annotators to select the references from the candidate sets based on two criteria. The criteria for selecting references are as follows: (1) the reference is found in at least two candidate sets and (2) it is relevant to the event/topic. We measured the similarity of the candidate sets in terms of the ROUGE (Lin Reference Lin2004) scores shown in Table 2. From Table 2, it is observed that LSA, Luhn, and Textrank extracted similar tweets compared to the Lexrank technique. The relevance of a tweet in a candidate set is measured according to the following procedure. We obtained the top n-grams (unigrams, bigrams, and trigrams) from each set of candidates and formulated queries based on the n-grams to rearrange the tweets. The distribution of n-grams in a particular cluster is shown in Fig. 9. Formally, we can represent this as an information retrieval (IR) problem described below.

Table 2.

Average ROUGE F-1 score for candidate summary similarity of Lexrank with other methods

Figure 8.

Gold reference summary creation process for the summarisation task.

Figure 9.

n-gram distribution in a particular cluster.

Let $q$ be a query of n-grams given over a set of documents $D$ where the IR task is to rank the documents in $D$ so that the documents most relevant to $q$ appear at the top. In our case, each tweet within a candidate set is a document and the particular set is the document pool which is represented as:

(17) \begin{equation} D=\bigcup _{k=1\cdots n }\tau _{k} \end{equation}

where $\tau _{k}$ is the top $k$ tweets retrieved. When $k=n$ , all the tweets from a cluster are retrieved. To obtain the ranking scores, we used the Sentence BERT (SBERT) (Reimers and Gurevych Reference Reimers and Gurevych2019) system, which measures the semantic similarity of the tweets in a given cluster. In SBERT, the cosine similarity between the query and the tweet is measured by computing the cosine of the query vector $u$ with the tweet vector $v$ . Both vectors $u$ and $v$ have the same dimensions. To rank the tweets, Spearman’s rank correlation is computed between the cosine similarity of the query and the tweets which is defined as:

(18) \begin{equation} \rho = 1 - \frac{\sum _{i=1}^{n}(u_{i}-v_{i})^2}{n(n^2-1)} \end{equation}

where $u_i$ and $v_i$ are the corresponding ranks of $u$ and $v$ , for $i = 0$ , $\cdots$ , $n -1$ . For example, in a particular cluster, the top trigram “mj akbar resigns” forms the query $q =$ “mj akbar resigns.” The query is then issued to SBERT to retrieve the top $k$ tweets from the given cluster. The annotators then pick the tweets from each candidate set based on informativeness, fluency, and succinctness of the contents and write the summaries with up to three sentences. We measured the agreement ratio between the annotators with standard ROUGE metrics using ROUGE-1.5.5 and obtained average F ROUGE-1 of 0.828, ROUGE-2 of 0.795, and ROUGE-L of 0.825. The summaries written by the two annotators are compared and manually aligned (through a consensus) to generate the final reference summaries.

4.4 P-Gen set-up

For the P-gen set-up, we used the pre-trained model pre-trained_model_tf1.2.1 Footnote ^m which is trained on the CNN/Daily Mail dataset. This model uses a vocabulary of 50K words with 21,50,1265 parameters. This model has 256-dimensional hidden states and 128-dimensional word embeddings. We set the parameters max_enc_steps and max_dec_steps to 400 and 10 tokens, respectively, to restrict the length of the source text and the summary, respectively. For beam search decoding, we set beam_size to 4.

4.5 Presumm set-up

From Presumm, we used the bertext_cnndm_transformer Footnote ⁿ pre-trained model for the abstractive summarisation task. This model is also trained on the CNN/Daily Mail dataset. Presumm uses Trigram blocking mechanism to prevent redundancy of phrases/sentences in the generated summary. The transformer decoder has 768 hidden units with a hidden size of 2,048 for all the feedforward layers. The original model adopted a two-stage fine-tuning approach. The model first undergoes fine-tuning using an extractive BERT model. This involves training the model to identify the most important sentences or phrases in the text that should be included in a summary. Essentially, the model learns to choose relevant content for summarisation from the larger body of text. After fine-tuning with extractive BERT, the model undergoes fine-tuning with an abstractive BERT model. In this stage, the model is trained to generate new concise summaries using its learnt knowledge of the source text. The abstractive model works to create summaries that are not just extracts from the original text but are rewritten and rephrased to be coherent and fluent. In our case, we fine-tuned only at the second stage. The following settings were applied during the training process: - drop_out = 0.2: A dropout rate of 20% was established to help prevent overfitting by randomly disabling a portion of the neurones during training. This regularisation technique improves the generalisability of the model. - beam_size = 5: A beam size of 5 was chosen for the beam search algorithm. This setting allows the model to consider multiple possible paths during the generation process, potentially improving the quality and diversity of the output. - lr = 0.02: A learning rate of 0.02 was used to control the speed of updates to the model’s parameters. This rate determines how quickly the model learns and adjusts its weights during training. - train_steps = 15000: A total of 15,000 training steps were specified to control the duration of the training process. This influences the performance and training time of the model. - batch_size = 200: A batch size of 200 was selected to indicate the number of samples processed in each training iteration. This choice affects the efficiency of training and the stability of the model’s learning process.

4.6 BART set-up

For the BART set-up, the bart-large-cnn Footnote ^o pre-trained model was utilised. Fine-tuning of the model was conducted with the following settings: - TOTAL_NUM_UPDATES = 20000: This specifies the total number of training iterations applied during the fine-tuning process. The model’s learning is influenced by this number, impacting the extent of its training. - UPDATE_FREQ = 4: Updates were applied after every four training steps. This approach allows the model’s learning to stabilise by accumulating gradients over multiple steps before applying them. - WARMUP_UPDATES = 500: During fine-tuning, 500 warm-up updates were conducted. This involved gradually increasing the learning rate from a lower initial value to its target value, which can help prevent sudden large parameter changes at the start of training. - LR = 3e-05: A learning rate of 0.00003 was set. This controlled the speed of parameter updates, allowing for careful adjustments to the model’s weights and preventing overly rapid changes. - BEAM_SIZE = 4: A beam size of 4 was used in beam search, an algorithm for decoding that explores multiple possible sequences simultaneously. This helps to improve the quality and diversity of the output by considering multiple paths during summary generation. Overall, these settings guide the fine-tuning process to enhance the performance of the pre-trained BART model on our specific summarisation task.

6.1 Clustering time complexity

It can be challenging to determine the precise time complexity of the k-means algorithm for the detection of events from tweets, as there are a variety of factors that can affect the calculation, such as the number of tweets, the number of features extracted from tweets, the number of clusters ( $k$ ), the convergence criteria, and the initialisation method used. The k-means algorithm is iterative in nature, and its time complexity is usually expressed in terms of the number of iterations ( $T$ ), the number of data points ( $N$ ), the dimension of the data ( $D$ ), and the number of clusters ( $k$ ). Therefore, the overall time complexity of the k-means algorithm is $O\left (T \times N \times k \times D\right )$ . In HAC, the algorithm first computes similarities or distances between tweets, which has a time complexity of $O\left (N^{2}\right )$ . This is because HAC needs to compare each tweet with every other tweet. HAC initialises each tweet as a cluster that has a time complexity of $O\left (N\right )$ . In each iteration, HAC merges the two closest clusters for $\left (N-1\right )$ iterations. This requires an update in the distances between the newly formed cluster and the remaining clusters. The time complexity for these updates depends on the distance metric and the data structure used to store the distances. If a priority queue or a suitable data structure is used, then each update might take $O\left (log N\right )$ time, leading to an overall complexity of $O\left (N log N\right )$ for agglomeration. Once all the clusters are merged into a single cluster, we have our final result. This step involves $O\left (N\right )$ operations. The most influential element of the process is usually the agglomeration step. Therefore, the total time complexity of HAC for the detection of tweet events is likely to be approximately $O\left (N^{2} + N log N\right )$ , depending on the exact implementation and the distance measure used. Estimating the time complexity of AP for event detection from tweets can be a difficult task, as it depends on a variety of elements, such as the magnitude of the input data, the parameters selected, and the implementation specifics. AP iteratively updates message passing values between data points to determine the best exemplars (cluster centres) for the data points. The main steps involved in the algorithm are to update the messages and update the responsibility/availability matrix with a time complexity of $O\left (N^{2}\right )$ , where $N$ is the number of data points. Following the message update step, the algorithm chooses exemplars based on responsibility and availability values. This step has a complexity of $O\left (N^{2}\right )$ . The number of iterations the algorithm needs can vary depending on the data and how it converges. In the worst case, the number of iterations could increase in relation to the number of data points, resulting in an overall time complexity of $O\left (N^{3}\right )$ .

For smaller datasets, k-means might perform faster due to its linear dependence on the number of data points. As the dataset grows, the quadratic dependence of AP on the number of data points can make it significantly slower than k-means. In general, the HAC time complexity has a clear dependence on the number of tweets and the chosen linkage criterion. On the other hand, the complexity of AP is more dependent on the convergence properties of the data and the algorithm parameters. Additionally, AP can discover clusters with irregular shapes, while k-means assumes that clusters are spherical and equally sized. The choice between these algorithms should also consider the quality of the clustering results and the suitability of the algorithm for the underlying data distribution and problem. On our dataset, both the k-means and HAC clustering algorithms did not generate appropriate clusters that represent the top-level events and the corresponding subevents. For example, the event “US Presidential Elections” has several sub-events such as (a) before the election, (b) on the day of the election, and (c) after the declaration of the election results. Both k-means and HAC clustering did not generate quality clusters to represent these subevents. This is because most of the clusters generated by k-means and HAC contain tweets representing almost all of the subevents. This has a significant impact on the summarisation task because a cluster consisting of multiple subevents may not be suitable for the appropriate summary generation.

6.2 Time complexity of summarisation

Comparing the time complexity of various text summarisation models can prove to be a difficult task, as it depends on various factors, including model architecture, input size, and hardware. The time complexity may also be subject to variations depending on how the model is implemented and optimised. The pointer network with coverage (P-Gen) has been introduced for the purpose of abstractive text summarisation. It is important to note that the time complexity of this model is primarily influenced by the length of the input document as well as the length of the output summary. In terms of complexity, it can be approximated as $O\left (n\right )$ , where $n$ represents the length of the input document. Presumm employs pre-trained encoder models, such as BERT, for both extractive and abstractive summarisation. The computational efficiency of these models is primarily based on the length of the input document and the number of tokens processed by the model. With regard to BERT-based models, computational efficiency is conventionally characterised by a time complexity of $O\left (L\right )$ , where $L$ denotes the number of tokens present in the input. BART is a model that utilises sequence-to-sequence technology for abstractive text summarisation. As with other models of this type, the level of complexity is linked to the lengths of both the input document and the output summary. Given its utilisation of multiple attention mechanisms and transformer layers, BART’s complexity tends to be greater than that of simpler models. In fact, its complexity can reach a level higher than $O\left (n\right )$ , sometimes reaching $O\left (n^{2}\right )$ or worse for long documents, depending on the particular implementation and model size.