Authorship attribution

Considerable previous research has evaluated computational approaches to authorship attribution (He et al. Reference He, Lashkari, Vombatkere and Sharma2024; Huang, Chen, and Shu Reference Huang, Chen and Shu2025; Stamatatos Reference Stamatatos2009; Swain, Mishra, and Sindhu Reference Swain, Mishra and Sindhu2017). Recently, many studies in this area have focused on distinguishing between human and LLM-generated texts; however, for this work our interest lies solely in distinguishing between texts written by human authors. Many lexical, character, syntactic, semantic, structural, n-gram, application-specific, and content-specific features have been used for authorship attribution, including word frequencies, sentence length, vocabulary richness, character types, parts-of-speech, and semantic dependencies (He et al. Reference He, Lashkari, Vombatkere and Sharma2024; Huang, Chen, and Shu Reference Huang, Chen and Shu2025; Stamatatos Reference Stamatatos2009; Swain, Mishra, and Sindhu Reference Swain, Mishra and Sindhu2017). These features are often used as inputs for a variety of statistical and machine learning methods including naive bayes classifiers, support vector machines (SVMs), logistic regression, recurrent neural networks (RNNs), long short-term memory (LSTMs), Siamese networks, decision trees and more (Huang, Chen, and Shu Reference Huang, Chen and Shu2025; Stamatatos Reference Stamatatos2009).

Since the introduction of transformer-based LLMs in 2017 (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), scholars have also explored their use for authorship attribution. BERT and its many variants have been applied to this task, often in combination with techniques like contrastive learning, gradual unfreezing and slanted triangular learning (Huang, Chen, and Shu Reference Huang, Chen and Shu2025). Recent work has also explored the ability of larger encoder–decoder (Hicke and Mimno Reference Hicke and Mimno2023; Najafi and Tavan Reference Najafi and Tavan2022) and decoder-only (Adewumi et al. Reference Adewumi, Habib, Alkhaled and Barney2024; Huang, Chen, and Shu Reference Huang, Chen and Shu2024; Huang, Murakami, and Grieve Reference Huang and Grieve2024; Wen, Guo, and Zhang Reference Wen, Guo and Zhang2024) models to perform attribution.

However, of these studies only Hicke and Mimno (Reference Hicke and Mimno2023) and Adewumi et al. (Reference Adewumi, Habib, Alkhaled and Barney2024) use literary data in their analyses. In addition, only Huang et al. (Reference Huang and Grieve2024) examine what features models use to attribute texts, and do so only by asking the models to self-report their reasoning. In contrast to most of these works, we do not study authorship attribution with the intention of creating or evaluating maximally effective attribution methods. Instead, our interest is in studying what signals of literary style exist at very small scales and to what extent LLMs can help us identify and analyze these signals. To this end, we focus on what LLM performance can tell us about literary style and about LLMs’ capabilities, including a series of ablation and probing experiments into what features the models use for attribution.

Genre identification

Genre identification is often more difficult to operationalize than authorship attribution due to the complexity of genre as a concept; unlike authorship, variable criteria may be used to define a genre and what texts belong to it, making it more challenging to establish what the “correct” label for a text is (Underwood Reference Underwood2016). However, despite these challenges, many computational approaches to genre identification have been proposed and evaluated. Some use bag-of-word features along with statistical and light-weight machine learning methods like logistic regression, Naive Bayes, SVMs and more (Allison et al. Reference Allison, Heuser, Jockers, Moretti and Witmore2011; Hettinger et al. Reference Hettinger, Reger, Jannidis and Hotho2016; Sharmaa et al. Reference Sharmaa, Hu, Wu, Shang, Singhal and Underwood2020; Underwood Reference Underwood2016; Underwood et al. Reference Underwood, Black, Auvil and Capitanu2013). Other works use similar statistical and ML methods along with additional features like emotion arcs (Kim, Padó, and Klinger Reference Kim, Padó and Klinger2017) or more in-depth linguistic annotations (Calvo Tello Reference Calvo Tello2021). Yet another branch of research on genre identification uses unsupervised clustering algorithms with bag-of-words features (Al-Yahya Reference Al-Yahya2018), social network features (Ardanuy and Sporleder Reference Ardanuy and Sporleder2015; Coll Ardanuy and Sporleder Reference Coll Ardanuy and Sporleder2014) or embeddings and topic classifications (Sobchuk and Šeļa Reference Sobchuk and Šeļa2024).

Recent research has also explored the effectivity of using transformer-based LLMs for genre identification. Much of this research has focused on non-literary definitions of genre (Kuzman Reference Kuzman2023; Kuzman, Mozetič, and Ljubešić Reference Kuzman, Mozetič and Ljubešić2023; Roussinov, Sharoff, and Puchnina Reference Roussinov, Sharoff and Puchnina2025; Uchida Reference Uchida2024); these studies have largely found model performance promising, although they have also identified some model weaknesses such as expanding to cross-domain data. However, two studies, Liu et al. (Reference Liu, Huang, Li, Sun, Wu and Zhang2020) and Bamman et al. (Reference Bamman, Chang, Lucy and Zhou2024), have explored using a range of LLMs for literary genre identification. Both studies look at $\sim $ 500 word or token chunks of literary texts. Liu et al. (Reference Liu, Huang, Li, Sun, Wu and Zhang2020) evaluate several statistical methods, including a Naive Bayes algorithm and a fine-tuned BERT model, for detecting the genre of paragraphs from novels that have a large proportion of genre-relevant keywords. Bamman et al. (Reference Bamman, Chang, Lucy and Zhou2024) similarly evaluate linear and logistic regression, fine-tuned BERT, RoBERTa and Llama-3 8b models, and few-shot prompted GPT-4o, Llama-3 70b, and Mixtral 8x22b models for detecting the genre of randomly selected novel passages. Liu et al. (Reference Liu, Huang, Li, Sun, Wu and Zhang2020) find that BERT performs the best out of all tested methods and Bamman et al. (Reference Bamman, Chang, Lucy and Zhou2024) find that the Llama-3 8b, GPT-4o and Llama-3 70b models are most effective. Again, these results suggest that LLMs show promise for genre identification tasks. However, no study except Bamman et al. (Reference Bamman, Chang, Lucy and Zhou2024) explores what features the models use to perform genre identification, and Bamman et al. (Reference Bamman, Chang, Lucy and Zhou2024) only ask the chat-enabled models self-report on distinguishing genre characteristics. Again, our work differs from those cited because its primary interest is in using LLM behavior to evaluate what signals of genre-level style exist in very short text segments, and not optimizing the model’s performance on this task.

Comparing models

All models and baselines achieve above random performance on authorship attribution and genre identification (Figure 2). This confirms both that signals of literary style exist in 20–50 word texts and that all tested methods are capable of recognizing these signals. In addition, all methods achieve much higher accuracy for genre identification than authorship attribution; this may suggest that genre is easier to identify in short texts, or may result from the difference in the number of possible classes between the datasets (5 vs. 27).

Figure 2. The overall accuracy of each model for (left) authorship attribution and (right) genre identification.

Note: Accuracy is separated into results for samples from novels included in training and samples from novels withheld from training. Results of a single run are reported and error bars represent the standard error bootstrapped over 1,000 iterations. The y-axis is sorted by model’s performance on samples from in-training novel.

All methods perform better on excerpts from novels partially included in training than excerpts from novels fully withheld from training (Figure 2). This implies two things: first, that the LLMs are not relying purely on memorization from pre-training, and, second, that the stylistic signals used by each method are partially novel-specific. However, the two largest and best performing models — llama-3-8b and flan-t5-xl — still outperform random baselines by $\sim 2.5\times $ for genre identification and $\sim 6.5\times $ for authorship attribution on samples from withheld novels, indicating that the models learn some generalizable stylistic traits.

These large, generative models outperform all other methods (Figure 2), with llama-3-8b achieving the highest overall accuracy on both tasks. While both baselines – the SVM and cosine delta – achieve impressive performance on both identification tasks, they are outstripped by the larger transformer-based models. For these LLMs, we find that model performance increases with size, although the performance increase per parameter decreases as size grows.

Probing memorization

The larger LMs’ high performance may be due to memorization of the texts and associated information during pre-training. To test whether llama-3-8b and flan-t5-xl are using memorized information, we adapt the prompt from Chang et al. (Reference Chang, Cramer, Soni and Bamman2023) (Figure 3). We use this prompt to query the base version of both models for the author of every excerpt in the authorship attribution and genre identification datasets, recording a match if the model’s unconstrained generation matches the correct author name. We use authorship attribution to probe for memorization in both datasets because the idea of genre is more generalizable than that of author. The only ways for a model to correctly report the author of a text are (a) if it has memorized the text and author or (b) if it is able to correctly match the text to a stylistic profile of an author. We assume that the base versions of these models are unlikely to have a stylistic profile of each author in our dataset, and therefore assume that they will only return the correct author for a high proportion of text segments if they have memorized the texts. In contrast, it seems more likely that the models have developed stylistic conceptualizations of genre, and therefore we deem probing for this trait a less effective method of testing for memorization.

Figure 3. The prompt used to probe llama-3-8b and flan-t5-xl for memorization of studied texts.

We find that llama-3-8b correctly provides the author of 33.11% (SE: 0.51) of excerpts from the authorship attribution dataset and 42.06% (SE: 1.13) of samples from the genre identification dataset, suggesting it has memorized many of the included texts. For samples from novels withheld from training, fine-tuning llama-3-8b only leads to a 13.31% increase in attribution accuracy. In addition, of the 5,021 excerpts from the authorship attribution dataset correctly attributed by the fine-tuned model, 2,177 (43.36%) are also correctly attributed by the prompted model. However, the prompted llama-3-8b model’s accuracy by author is uncorrelated with the accuarcy by author of the fine-tuned model ( $p=0.2$ ). For example, the prompted model correctly identifies the most excerpts by Charles Dickens (88.45%), but the fine-tuned model’s performance on Dickens is among its worst (44.22%); additionally, of the 134 samples correctly identified by the fine-tuned model, 133 are also correctly identified by the prompted model. This may suggest that, throughout fine-tuning, some of the information previously memorized by the model is lost (thus the drop in accuracy for Charles Dickens) and that capacity is instead used to store information on the authors for whom no information was memorized (so the accuracy for Marie Corelli goes from 1.3% to 62.05% with fine-tuning).

To determine whether llama-3-8b’s memorization of texts is correlated to author popularity, we borrow a proxy measure of author popularity – the length (in characters) of their Wikipedia page – from D’Souza and Mimno (Reference D’Souza and Mimno2023). We find that there is, indeed, a strong positive correlation between the popularity of an author and the prompted model’s accuracy at attributing excerpts from that author for both the authorship attribution dataset (Pearson’s R: 0.65, $p < 5\times 10^{-4}$ ) and the genre identification dataset (Pearson’s R: 0.60, $p < 5\times 10^{-4}$ ). However, no significant correlation exists between author popularity and the fine-tuned model’s performance on the authorship attribution dataset ( $p = 0.75$ ) or author popularity and the fine-tuned model’s ability to provide the correct genre of an excerpt by that author ( $p=0.56$ ). Furthermore, we find a correlation between author popularity and the proportion of excerpts correctly attributed by the fine-tuned llama-3-8b model that are also correctly attributed the prompted model (Pearson’s R: 0.58, $p < 5\times 10^{-3}$ ). Thus, it appears that the fine-tuned model maybe relying on memorized information when attributing quotes from popular authors and information learned through fine-tuning when attributing quotes from less well-known writers.

In contrast to llama-3-8b, flan-t5-xl only correctly provides the author of one sample from the authorship test dataset (0.01%, SE: 0.01) and four samples from the genre test dataset (0.22%, SE: 0.01). These correctly attributed excerpts are all from relatively popular authors, but so few exist that no conclusive insights can be drawn from these results. As with llama-3-8b, there is no significant correlation between the popularity of authors in the authorship attribution dataset and the fine-tuned flan-t5-xl model’s accuracy at attributing excerpts from these authors ( $p=0.71$ ) or the fine-tuned flan-t5-xl model’s ability to provide the correct genre of excerpts by these authors ( $p = 0.95$ ).

Because of the stark differences between the behavior of llama-3-8b and flan-t5-xl when probed for memorization, we hypothesize that they use different information to perform style identification: while llama-3-8b has access to memorized information from pre-training which it may be leveraging, flan-t5-xl appears to be learning stylistic representations from fine-tuning.

We further evaluate whether the base llama-3-8b and flan-t5-xl models are able to predict the genre of excerpts from the genre identification test dataset. While we take accurately reporting the author of an excerpt to indicate memorization of text, this is not necessarily true for genre; models may have an internal representation of genre that can be applied without fine-tuning. For this probe, we again adapt the prompt from Chang et al. (Reference Chang, Cramer, Soni and Bamman2023) (Figure 4). Since it is difficult to define a “correct” genre in an unconstrained setting, we include a list of possible genres in the modified prompt. We find that llama-3-8b and flan-t5-xl are able to correctly report the genre of 56.89% (SE: 1.18) and 44.39% (SE: 1.17) of excerpts respectively, suggesting that these models do have internal representations of genre that can be leveraged.

Figure 4. The prompt used to probe llama-3-8b and flan-t5-xl for internal representations of genre.

There are not significant correlations between the accuracy per genre of the fine-tuned and prompted llama-3-8b models ( $p=0.16$ ) or flan-t5-xl models ( $p=0.54$ ). Examining the confusion matrices of responses from the fine-tuned and prompted versions of both models (Figures 5 and 6), we see that both prompted models tend to incorrectly label excerpts as historical fiction or fantasy, while the fine-tuned models do not. The prompted llama-3-8b model also has a strong tendency to mislabel excerpts as horror whereas the prompted flan-t5-xl model tends to mislabel excerpts as mystery. Both fine-tuned models also misidentify some excerpts as horror and mystery, although the extent to which these labels are assigned to misattributed samples does not follow the same pattern as the prompted models. Science fiction is the label to which the least samples are mis-assigned in almost all cases, suggesting there is something unique about the genre. Overall, we see that there are some trends in how the models treat genre, although these trends are not broadly generalizable.

Figure 5. Confusion matrices of the responses of the prompted (left) and fine-tuned (right) llama-3-8b models for genre identification.

Note: Correct labels are represented by rows and model outputs are columns; labels produced outside of the correct set are ignored. The rows sum up to $\sim $ 100%.

Figure 6. Confusion matrices of the responses of the prompted (left) and fine-tuned (right) flan-t5-xl models for genre identification.

Note: Correct labels are represented by rows and model outputs are columns. The rows sum up to $\sim $ 100%.

Because the prompted flan-t5-xl model could not provide the authors of these texts, it seems unlikely that the genres of the identified excerpts were memorized at the book level; instead, we hypothesize that the model is able to recognize signals of genre-level style without fine-tuning. This may contribute to the models’ much higher performance on genre identification. Fine-tuning only improves flan-t5-xl’s accuracy at genre identification on out-of-training excerpts by 14% and only improves llama-3-8b’s by 3.67%. Again, this suggests that flan-t5-xl is “learning” more from fine-tuning than quantized llama-3-8b, although both models have a leveragable internal representation of genre before fine-tuning.

Analysis by class

Examining results from the fine-tuned models and baselines, we see that the accuracy of each model varies more by author than by genre (Figure 7). Variation by class tends to lessen as the model size increases, although the difference in accuracy by author for llama-3-8b is greater than for flan-t5-xl. There is some agreement between models as to which authors are easier to classify – for example all models but llama-3-8b perform comparatively very well on samples from Frances Hodgson Burnett and Joseph Conrad – but this is not the case for all authors or all models. There are more consistent patterns across models for genre; science fiction is the class on which each model except llama-3-8b performs best or second-best and horror is the least-identified class for the SVM and all transformer-based models larger than flan-t5-base. It is not clear why these trends exist; they may be artifacts of model performance or indicate that these classes are somehow more or less stylistically distinct than others. Overall, we find that for both genre and authorship, the difference between class-level accuracy is considerably more extreme for samples from novels withheld from training, perhaps suggesting that the model’s ability to recognize author- or genre-level, and not just novel-level, signals of literary style is more variable (Appendix E.1: Figures E1 and E2).

Figure 7. Accuracy by author (left) and genre (right) for each model.

Note: Results of a single run are reported and error bars represent the standard error bootstrapped over 1,000 iterations. The x-axes are sorted by the accuracy of llama-3-8b.

Previously, Hicke and Mimno (Reference Hicke and Mimno2023) found that LLMs often used 2–3 authors as “scapegoats” when fine-tuned for authorship attribution, assigning the majority of misattributed samples to them. We replicate this finding, showing that llama-3-8b scapegoats more than any model except flan-t5-small (Figure 8 (left)). Although the extent to which authors are scapegoated is not consistent across models, we find that George Gissing is among the top two most scapegoated authors for each transformer-based model. Hicke and Mimno (Reference Hicke and Mimno2023) showed that authors with larger vocabularies and less unique language usage were more likely to be scapegoated; however, while Gissing does have the fourth lowest uniqueness score of all authors in the dataset (63.69, range: 53.02–113.88, values calculated as in Hicke and Mimno (Reference Hicke and Mimno2023)), he also has the fourth-smallest vocabulary (6,468, range: 5,152–8,189). Hicke and Mimno (Reference Hicke and Mimno2023) also hypothesized that more famous authors may be less scapegoated, as models have more information about their writing before fine-tuning; we find that Gissing is indeed the sixth least famous author by Wikipedia length (27,013, range: 11,532–140,276). Thus, we again find evidence that less-famous authors with less-unique vocabulary usage may be more likely to be consistently scapegoated, although other factors are clearly impactful.

Figure 8. The % of misattributions assigned to the top n “scapegoated” authors (left) or genres (right).

Note: Results are reported for a single run and error bars represent the standard error bootstrapped over 1,000 iterations.

In contrast to the authorship attribution results, the only model to show evidence of extreme scapegoating for genre identification is llama-3-8b (Figure 8 (right)). For llama-3-8b, $\sim $ 30% of misattributed samples are labeled as fantasy, horror and mystery respectively, whereas only $\sim $ 3% of misattributed samples are labeled as science fiction or historical fiction. There is not a large variation in genre uniqueness (range: 70.92–81.83) that explains this difference and science fiction and historical fiction have the two largest vocabularies (11,369 and 10,407, range: 8,892–11,369). Perhaps the larger vocabularies of these two genres represent the usage of situation-specific terminology (e.g., relating to spaceships or duels) that is unlikely to appear outside of the genre, unlike the use of broader but generally applicable vocabulary by authors which may have caused the opposite trend to appear in Hicke and Mimno (Reference Hicke and Mimno2023).

Overview

From the classification experiments, we find that it is possible to detect multiple kinds of literary style in short text segments using LLMs. The largest, generative LLMs – llama-3-8b and flan-t5-xl – perform the best on both attribution tasks, but variations in these models’ performances reveal differences in what signals they use to recognize literary style. llama-3-8b appears to rely partially on memorized information, particularly about works from more popular authors, while flan-t5-xl demonstrates very little memorization. We also find differences between how genre- and author-level styles are identified. Because the base flan-t5-xl model is able to produce the genre of many excerpts for which it cannot provide the author, it seems that signals of genre-level style are better known by LLMs even without memorization or fine-tuning; this makes sense, as genre is a much more generalizable concept than author. In addition, differences in the class-level accuracy of models fine-tuned for genre identification vs. authorship attribution and in their scapegoating suggests further differences between genre- and author-level style, or in how these styles are identified by LLMs.

Comparing perturbations

We find that the effects of each text modification become clearer in larger models and that these effects are relatively consistent across authorship attribution and genre identification (Figure 9). However, the Flan-T5 and Llama models’ accuracy drops slightly when capitalization and punctuation are removed in for authorship attribution, but these disruptions barely affect or improve these models’ performance for genre identification. These results indicate that capitalization and punctuation patterns may carry information about authorship, but do not indicate genre; this makes intuitive sense as authors seem more likely develop specific punctuation of capitalization habits than entire genres. The performance of models degrades somewhat when proper nouns are removed for both tasks, but this effect is relatively small for samples from withheld novels (on average 9.28% for llama-3-8b and 4.02% for flan-t5-xl).

Figure 9. The accuracy of each model on samples from withheld novels for authorship attribution (left) and genre identification (right) across each text perturbation.

Note: Results are reported for a single run and error bars represent the standard error bootstrapped over 1,000 iterations.

For all of the LLMs, the most impactful modifications are masking stop words and shuffling word order. In fact, when word order is shuffled the performance of flan-t5-xl drops below that of the SVM, a bag of words model, for both stylistic identification tasks. This demonstrates how heavily reliant the models, particularly the larger models, are on sequence-level information and how strongly their representations of style rely on word order.

The apparent impact of stop words initially appears consistent with the findings of previous stylometric studies tracing back to Mosteller and Wallace (Reference Mosteller and Wallace1963), which argue that common function words play a large role in defining authorial style. To further explore this phenomenon, we evaluate the two highest performing models on an alternate perturbed dataset where, instead of masking stop words, we mask a number of random words in each sample equal to the number of stop words in that sample. Performance on this dataset is lower for both tasks and both models (by $\sim $ 8% for llama-3-8b and $\sim $ 0.5% for flan-t5-xl on withheld data) than when only stop words are masked. Therefore, while stop words clearly contribute to distinguishing authorial and genre-level style, as past research has suggested, these results indicate that LLMs make equal or greater use of less-frequent words.

Because stop words belonging to some parts-of-speech appear more frequently than others, we next look at the relationship between the average number of masked words per sample and model accuracy for each stop word variant. For authorship attribution in llama-3-8b and flan-t5-xl, there is a significant negative relationship (all Pearson’s R $< -0.9$ , $p < 5\times 10^{-4}$ ) between the number of words masked and model accuracy for both out-of-training and in-training novel samples (Figure 10). This indicates that the differences in model accuracy between each stop word variant are largely due to the number of stop words being masked and not the importance of certain types of stop words for authorship attribution.

Figure 10. llama-3-8b (top) and flan-t5-xl’s (bottom) accuracy when stop words of a certain part of speech are masked by average number of those stop words per sample for both tasks.

Note: Results are reported for a single run and error bars represent the standard error bootstrapped over 1000 iterations. The pronoun data are labeled to provide context for in-text analysis.

However, for genre identification in both models there is only a significant relationship between the number of words masked and variant accuracy for novels in training (Pearson’s R $< -0.7$ , $p < 0.03$ ). Therefore, we hypothesize that some stop word variants do have a disproportionate effect for models when generalizing to genre-level style; in particular, we see that pronouns have an unexpectedly large effect for flan-t5-xl whereas conjunctions have an unexpectedly large impact and pronouns and prepositions have an unexpectedly small effect on llama-3-8b (Figure 10).

Overall, regardless of correlation with the number of words masked, we find that pronouns have the largest effect on flan-t5-xl on both tasks for both samples from novels in-training and out-of-training (Figure 10). Pronouns also have a large impact on the bag-of-words baselines; they have the largest effect on performance on both tasks for the SVM and on authorship attribution for cosine delta (Figure 11). In contrast, for llama-3-8b pronouns do not have a disproportionate effect on accuracy; they are no more or less impactful than any other stop word category conditioned on the number of words masked. However, this may be because llama-3-8b is relying on memorized information and therefore masking stop words primarily impacts the model’s ability to match inputs with memorized texts. Because pronouns disproportionately affect the models which have not shown signs of memorized information, we hypothesize that they are important for detecting literary, particularly authorial, style.

Figure 11. The SVM (top) and cosine delta’s (bottom) accuracy when stop words of a certain part of speech are masked by average number of those stop words per sample for both tasks.

Note: The pronoun data are labeled to provide context for in-text analysis.

Overview

The ablation experiments allow us to infer information about the relative importance of several syntactic and semantic text properties. We find that some stylistic choices, like capitalization and punctuation, affect the more fine-grained authorial style but have less impact on the more generalized genre-level style. Proper nouns have little impact overall on models’ abilities to identify texts from out-of-training novels, likely because they are usually novel-specific. On the other hand, word order proves to be very important in identifying both forms of literary style, especially for flan-t5-xl. Finally, we find evidence that stop words are not as important for distinguishing literary style as previous work has suggested; despite this, for models that do not show evidence of text memorization pronouns appear to be impactful in characterizing literary, particularly authorial, style.

Training influence

Our first internal probe attempts to discern which input features, here words, most influence the style-detection models. We hypothesize that the words with the greatest impact on model behavior will carry the most information about style. To measure the impact of words, we use Shapley values (Lundberg and Lee Reference Lundberg and Lee2017), a metric from game theory that measures the contribution of each input feature to an outcome. Specifically, we use a gradient-based implementation for calculating Shapley values available through the inseq package (Sarti et al. Reference Sarti, Feldhus, Sickert, van der Wal, Nissim and Bisazza2023) to examine the impact of each word in the input on each token in the output representing an author name or genre. We strip punctuation and capitalization from features during analysis and remove tokens from the input prefixes (GENRE: <extra_id_0> and AUTHOR: <extra_id_0>). We then look at the most impactful words by mean and summed Shapley value over all excerpts in the test datasets.

Unfortunately, the resulting Shapley values do not provide clearly interpretable information about style. The most prominent words by mean Shapley value appear very rarely in the input texts and consist primarily of proper nouns and other uncommon words (Appendix E.2). While this informs us that rare or uncommon words can be highly impactful in attributing the individual samples containing them, it does not allow us to draw broader conclusions about what features flan-t5-xl makes use of. In contrast, the most impactful words by summed Shapley value appear to be the most frequent words in the corpus. Many pronouns appear among the most impactful words by summed Shapley value for both genre indentification and authorship attribution – more than any other stop word – but this may simply reflect their frequent usage. Overall, the Shapley values demonstrate that words used at a variety of frequencies can be impactful for style identification, but do not provide further insight.

Cross attention

Our second probe examines the cross attention paid by each token representing an author name or genre (from here on answer tokens) in an output to every token in the input sample. Here, we again hypothesize that the most influential words in the input will hold the most information about style, and use attention as a proxy for influence. For every sample of interest, we create versions of the output reporting that it belongs to every class (so, for example, a sample from a horror novel will have output versions created reporting that it is from each of the five genres). Then, we extract the attention paid by the answer tokens representing each correct or incorrect class to each input token. We analyze the tokens with the largest summed and mean attention values across all correct and incorrect answers.

As with Shapley values, the highest mean attention is paid to rare or unusual words, while the most frequently used words rank highly by summed attention (Appendices E.3.1 and E.3.2). In order to extract more interpretable results from these values and compare cross-attention values, we use a modified version of the fightin’ words algorithm (Monroe, Colaresi, and Quinn Reference Monroe, Colaresi and Quinn2008),Footnote ⁷ a Bayesian text analysis method that reports which words distinguish two corpora. To adapt this algorithm for use with attention values, we replace the standard matrix of word counts with a matrix of summed attention values. Because samples with incorrect answer tokens are evaluated 26 times (authorship attribution) or 4 times (genre identification) more frequently with than those with correct answer tokens, we standardize the summed attention values from these samples.

We first use the modified fightin’ words algorithm to compare the summed attention values for each class to the values for all other classes in that dataset (e.g., Charles Dickens vs. all other authors). Then we re-rank the input tokens by their fightin’ words scores to find the twenty most and least distinctive words for each genre or author (Appendices E.3.1 and E.3.2). Next, we count which token types feature frequently in these lists, counting a token each time it appears and allowing for double counting. From this analysis, we find that punctuation makes up 16% of the most distinctive tokens for authors and 12% for genre. Again, these stylistic elements appear to be slightly more important for distinguishing authorial style than genre-level style. Stop words make up 26.5% and 23.7% of the most distinctive tokens for authors and genres respectively. For both genre identification and authorship attribution, pronouns appear in these lists the most of any stop word type (11.5% and 12.96%), once more reinforcing their importance. Proper nouns also appear frequently in the distinctive word lists for authors and occasionally for genres; however, these results may be affected by excerpts from in-training novels, where proper nouns may be significant aids in attribution.

The pronouns in the distinctive word lists for each author seem to most frequently distinguish between commonly used points of view (e.g., first person vs. third person). However, they also appear to provide information about the subjects and characters the authors frequently discuss. We use the fightin’ words algorithm to run a further comparison of the attention paid to input tokens by answer tokens representing correct female and male authors. The two tokens that most strongly characterize female authors’ language by this metric are _her and _she; the token _She is the only other pronoun to appear in the list of fifty most distinctive words for female authors. In contrast, the only pronouns to appear in the set of fifty most distinctive tokens for male authors are _we, _us, _our and _his. These results demonstrate that female pronouns are more stylistically important for attributing texts from female authors, perhaps suggesting that female authors more frequently discuss women or do so distinctly from male authors. The results also show that first-person plural pronouns are similarly stylistically significant for attributing texts from male authors.

Next, we use the fightin’ words algorithm to compare the attention paid to input tokens when the correct and incorrect classes are represented by answer tokens. For the authorship attribution task, we see that proper nouns and common subword tokens appear frequently in the fifty most distinctive tokens for correct authors (Appendix E.3.1). The only stop word present in this list is _the. In contrast, for genre identification, the fifty tokens most distinctive of correct answers are mostly punctuation, stop words and simple subword tokens (Appendix E.3.2). _the is the most distinctive word for correct genres. While the word _the is unlikely to ground compelling theories of genre by itself, it may be standing in implicitly for features like the use of concrete nouns.

The tokens most associated with incorrect answers for both authorship attribution and genre identification contain many capitalized words, which likely represent the the first words in samples. Many of these are stop words. This could indicate that the fine-tuned flan-t5-xl models pool attention in the first word of a sentence when confused or uncertain; this hypothesis is supported by the results of Xiao et al. (Reference Xiao, Tian, Chen, Han and Lewis2023), which show that the first words of sequences act as attention sinks in decoder-only models during dialogue exchanges.

These results again suggest that there are differences in authorial and genre-level style. Proper nouns and specific topical words appear to be more impactful in forming authorial style, whereas more common words and small subword tokens are comparatively more important for characterizing genre-level style. Perhaps this is because, while authors use a relatively set vocabulary in their works, the same is not true for a genre made up of works by many authors with varied vocabularies. Thus, in distinguishing genre, the presence of very specific terms may not be a reliable signal, whereas the frequency with which particular sets of more common words are used plays a larger role.

Contextual embeddings

The previous probes have treated words as unigrams. In our third probe, we explore whether flan-t5-xl’s understanding of style is contextual; we examine whether how words are used matters as well as which words are used. We hypothesize that both vocabulary and word usage will contribute to authorial and genre-level style.

To test this hypothesis, we model the language of each class (author or genre) by creating average embeddings representing how words are used by that class. We pass every excerpt from the training datasets through the base flan-t5-xl model and create a contextual embedding of each word in that excerpt by averaging the embeddings of the word’s component tokens from the last four layers of the model’s encoder stack. We then create a class-level embedding for each word by averaging the embeddings of each instance of the word from that class in the training dataset. This results in a dictionary for each class containing an average embedding representing each word in the training dataset vocabulary. We create a representation of the “average” language usage in the training dataset by averaging across the class-level representations of a word for every class that used that word.

To compare how unique each class’s language usage is, we find the average cosine similarity between the class’s usage of each word and the training dataset’s average usage of that word, adding zero when a word in the training dataset vocabulary was never used by the class. Similarly, we compare the language usage in a class’s test data to its training data by finding the average cosine similarity between each word’s embedding in a test sample and the embedding of that word created from the class’s training data and summing this average cosine similarity for each excerpt in the class’s test data.

We find that, for authorship attribution, the fine-tuned flan-t5-xl model performs better the more unique an author’s language is, compared to the training dataset norm, and the more similar their samples in the test dataset are to their samples in training. Specifically, we find a significant negative relationship between author-level accuracy and the similarity of an author’s word usage in training to the average training dataset language (Pearson’s R: -0.57, $p < 5\times 10^{-3}$ ) and a significant positive relationship between author-level accuracy and the similarity of the author’s test and training word embeddings (Pearson’s R: 0.66, $p < 5\times 10^{-4}$ ). These correlations suggest that contextual word usage does help flan-t5-xl to distinguish between authorial styles; more distinct authors by this metric are easier to identify.

However, there are no significant correlations between genre-level accuracy and similarity between average training embeddings and genre-specific training embeddings ( $p> 0.6$ ) or similarity between genre-specific training and test embeddings ( $p> 0.8$ ). This indicates that contextual word usage may do less to differentiate genres than authors; perhaps, this is again because genres are contributed to by many authors who may all have distinct word usage patterns.

Overview

From our probing experiments, we find that both frequently and infrequently used words can be useful for distinguishing between literary styles. Whereas uncommonly used words may be more individually powerful in determining authorial or genre-level style, common words that carry small stylistic signals can become significant over a corpus. We also find further evidence of differences between genre-level and authorial style. The results of the cross-attention and contextual embedding probes suggest that more uncommon words and words’ contextual usages are impactful in characterizing authorial style, but not genre-level style. These features may be less impactful for genre-level style because genres are made up of works by a large variety of authors who all use different vocabularies in different ways. Finally, we again find evidence that pronouns help to distinguish literary styles, both between authors and between male and female authors.