Plain language summary

This study applies computational methods to revisit the Formalist theory of defamiliarization, which posits that art renews perception by rearranging familiar forms in unpredictable ways. The research goal was to formalize and measure the stylistic tension between predictability and surprise using a large language model as a stand-in for a historically situated reader. To achieve this, the study employed a multi-stage experimental design. First, a GPT-2 style transformer model (223M parameters) was pre-trained on a large, high-quality corpus of modern Chinese text (FineWeb Edu Chinese V2.1) to establish a neutral linguistic baseline. This model was then fine-tuned for five epochs exclusively on the Selected Works of Mao Zedong to simulate a reader becoming saturated with “Maospeak,” the highly formulaic and militant language of the Mao era. The core metric was perplexity, which quantifies the model’s predictive uncertainty, or “surprise,” at each token. The primary finding was a dual process of familiarization and defamiliarization. By tracking the largest decreases in perplexity, the study identified the core phraseology of Maospeak, including canonical lists of class enemies and rigid rhetorical patterns. This “cognitive overfitting” demonstrated how propaganda automates language to create a predictable, low-perplexity environment and legitimatize violence. $N$ -gram entropy analysis confirmed that the Mao corpus was measurably more repetitive than a literary corpus at smaller $n$ -gram sizes, i.e., in its foundational vocabulary and short collocations. Concurrently, as the model specialized in Maospeak, its average perplexity on a corpus of 100 modern Chinese novels increased, making literary language appear more alien. Microscopic analysis of novels by Zhang Wei, Lilian Lee and Dung Kai-cheung revealed how literary style creates “non-anomalous surprise” through high-perplexity moments, such as creative metaphors, the use of classical Chinese syntax and the abrupt insertion of non-standard Chinese languages such as Cantonese. Ultimately, the research proposes “cognitive stylometry” as a new mode of inquiry, redefining style as a measurable cognitive signature that orchestrates a reader’s predictive processing by balancing familiar grounding with moments of meaningful deviation.

Introduction: Rethinking defamiliarization in the age of LLMs

Large language models (LLMs) present an exciting opportunity for literary theory, inviting new perspectives on the foundational concepts, such as meaning, attention and memory. Yet, in addition to ethical concerns (Weidinger et al. Reference Weidinger, John and Rauh2021; Hendrycks, Mazeika, and Woodside Reference Hendrycks, Mazeika and Woodside2023; Kirschenbaum Reference Kirschenbaum2023; Yan et al. Reference Yan, Sha and Zhao2024), their integration within humanistic studies faces two disciplinary challenges. On the one hand, radically anthropocentric beliefs continue to frame machine intelligence through a lens of inadequacy, as seen in Noam Chomsky’s critique of ChatGPT’s “false promise” (Chomsky Reference Chomsky2023) or Michele Elam’s assertion that the “creative process cannot be reduced to a prompt” (Elam Reference Elam2023). On the other hand, computational approaches that are adopted by literary scholars risk veering so far into benchmarking and quantification that they lose sight of literature as a source of aesthetic experience and interpretive engagement. These concerns are particularly relevant to the computational analysis of literary style, or stylometry, where the focus on precision metrics often overshadows the very texts under study. Paradoxically, highly accurate attribution models might fail to tell us anything about style.

An alternative path may lie in restarting the conversation between probabilists and humanists that unfolded in the early twentieth century but then subsided under historicist pressures. In 1913, Andrey Markov analyzed 20,000 letters from Pushkin’s poem Eugene Onegin to demonstrate his theory of dependent trials, later known as Markov chains, proving that the probability of a letter being a vowel or a consonant depends on the class of the preceding letter (Markov Reference Markov2006). Decades later, Claude Shannon reframed this problem in the empirical terms of information theory: “How well can the next letter of a text be predicted when the preceding N letters are known?” (Shannon Reference Shannon1951, 50). He demonstrated that humans possess an implicit knowledge of language statistics, effectively casting them as predictive language models. The aesthetic implications of this tension between predictability and surprise were anticipated by the Formalist scholar Viktor Shklovsky, who argued that art combats the “automatization of perception,” a process whereby, through habit, life “becomes nothing and disappears” (Shklovsky Reference Shklovsky and Berlina2017, 80). By contrast, in a process he called defamiliarization (ostranenie), art “increases the complexity and duration of perception” and “recovers the sensation of life” by rearranging familiar forms in unpredictable ways (Resseguie Reference Resseguie1991; Steiner Reference Steiner1984).

Recent advances in computational hardware, artificial intelligence and cognitive science call for a renewed humanistic perspective on these complementary phenomena. Trained on vast corpora spanning literature, journalism and social media, LLMs represent what Shklovsky could call the “familiar,” a statistically distilled reflection of human knowledge, language use and cultural expectations. Meanwhile, the concept of “defamiliarization” has been reinvigorated with new empirical evidence, from eye-tracking experiments showing that readers fixate longer on unexpected words (McDonald and Shillock Reference McDonald, Shillock, Carreiras and Clifton2004; Bonhage et al. Reference Bonhage, Mueller and Friederici2015) to fMRI scans revealing heightened mental engagement with unusual sentence structures, consistent with theories of the brain as a predictive organ (Dehaene et al. Reference Dehaene, Cohen and Sigman2005; Schuster et al. Reference Schuster, Weiss and Hutzler2025; Henderson et al. Reference Henderson, Choi and Lowder2016; Bohrn et al. Reference Bohrn, Altmann and Lubrich2012; Clark Reference Clark2013, Reference Clark2016). Shannon’s blurring of the human/machine reader distinction has also been pushed further, with research suggesting parallels between narrative comprehension in humans and in Transformer-based models (Caucheteux and King Reference Caucheteux and King2022; Kumar et al. Reference Kumar, Sumers and Yamakoshi2024), though scholars urge caution (Feghhi et al. Reference Feghhi, Hadidi and Song2024; Soni et al. Reference Soni, Srivastava and Kording2024; Fodor, Murawski, and Suzuki Reference Fodor, Murawski and Suzuki2025). Humanists are particularly well-positioned to contribute to this interdisciplinary dialogue – not to surrender to the discourse of technoscience, but to offer new perspectives on literature as a source of multi-layered engagements with our cognitive, emotional and predictive faculties (Kurzynski Reference Kurzynski2025), a goal that motivates this project.

Methods: Predictability and perplexity

The tension between expectation and surprise, central to both information theory and the notion of defamiliarization, can be formalized using perplexity, a measure of a statistical model’s predictive certainty (Jelinek et al. Reference Jelinek, Mercer and Bahl1977). Lower perplexity on the correct output indicates high confidence, while higher perplexity signals surprise. Most of the currently available LLMs learn to minimize this surprise by iterating over large amounts of text, either by predicting the next token in a sequence (autoregressive models) or by reconstructing an original sequence from a perturbed one (masked language models) (Paaß and Giesselbach Reference Paaß and Giesselbach2023, 19–78).

In this study, perplexity will serve as a metric for simulating human engagement with texts. While acknowledging the significant differences between natural cognition and artificial intelligence, notably the different ways in which they emerge (Lillicrap et al. Reference Lillicrap, Santoro and Marris2020), I leverage language models as proxies of historically situated human readers to revisit Shklovsky’s argument (Varnum et al. Reference Varnum, Baumard and Atari2024). Style, in this framework, is not merely a property of the text itself but is understood in relation to the predictive coding mechanisms inherent to both biological and artificial cognition. Phrases that elicit high perplexity will be especially relevant here, aligning with the understanding of style as a “deviation from the norm” (Enkvist, Spencer, and Gregory Reference Enkvist, Spencer and Gregory1964; Herrmann, Dalen-Oskam, and Schöch Reference Herrmann, van Dalen-Oskam and Schöch2015), whereas highly predictable phrases will instead reveal the automatized linguistic background against which stylistic distinctions emerge. Of particular importance to this article, however, is how an initially unfamiliar phrase becomes familiar through repeated exposure, and, conversely, how a particular phraseology becomes surprising when a model overfits on a different linguistic style. Such a process may offer a compelling computational analog for phenomena of broader cognitive-historical interest: for instance, how literary canons form, or how, under specific historical circumstances, prescriptive linguistic norms violently narrow the scope of legitimate expression (Bloch Reference Bloch1975; Epstein Reference Epstein1990; Ji Reference Ji2004; Hughes Reference Hughes2009; Klemperer Reference Klemperer and Brady2013).

Stylometrists have rarely considered predictability a primary stylistic feature, traditionally favoring more tangible elements like function words, syntactic structures and thematic profiles (Lutosławski Reference Lutosławski1898; Mosteller and Wallace Reference Mosteller and Wallace1963; Burrows Reference Burrows1987; Herrmann, Dalen-Oskam, and Schöch 2015; Rybicki, Eder, and Hoover Reference Rybicki, Eder, Hoover, Crompton, Lane and Siemens2016; Evert et al. Reference Evert, Proisl and Jannidis2017). This has changed with the advent of LLMs, whose human-like output spurred an urgent search for markers to differentiate machine from human text, with perplexity emerging as a key candidate. An influential early paper by Holtzman et al. argued that human text is not the most probable text: “natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens” (Holtzman et al. Reference Holtzman, Buys and Du2020). Subsequent studies confirmed that low-perplexity sequences often indicate machine provenance (Mitchell et al. Reference Mitchell, Lee and Khazatsky2023; K. Wu et al. Reference Wu, Pang and Shen2023). However, as newer LLMs produce text that is “nearly indistinguishable from human-written text from a statistical perspective” (Chakraborty et al. Reference Chakraborty, Tonmoy, Zaman, Bouamor, Pino and Bali2023), researchers have developed methods like watermarking, which embeds statistically detectable patterns into AI-generated content for provenance verification (Zhao et al. Reference Zhao, Ananth and Li2023; Kirchenbauer et al. Reference Kirchenbauer, Geiping, Wen, Krause, Brunskill and Cho2023). A few studies in the computational humanities have also begun to explore perplexity’s stylometric potential, using it to distinguish canonical from non-canonical novels (Y. Wu et al. Reference Wu, Bizzoni and Moreira2024), machine from human translation (Bizzoni et al. Reference Bizzoni, Juzek and España-Bonet2020) and political discourse from novelistic prose (Kurzynski Reference Kurzynski2023, Reference Kurzynski2024), signaling a broader turn toward evidence of cognitive and informational processing in literary analysis.

Datasets and tokenizers

The foundation of this study is the Fineweb Edu Chinese V2.1, a large-scale, high-quality Chinese corpus constructed from diverse datasets, including Wudao, Skypile, IndustryCorpus and Tele-AI (Yu et al. Reference Yu, Dai and Wang2025). The Fineweb authors employed a filtering pipeline where a scoring BERT model evaluated each document on a 0-to-5 scale for educational value and writing quality. The present study utilizes the “4–5” split, which contains only the highest-scoring, deduplicated content. From this top-tier partition of approximately 70 GB, one million Chinese passages varying in length from 256 to 1,024 characters were sampled for pre-training, serving as a neutral linguistic environment. Passages in languages other than Chinese as well as those with a high percentage of numbers and scientific notation were removed.

Two distinct test corpora were then constructed to represent stylistically opposed domains. The first is a seven-volume compilation of the Selected Works of Mao Zedong 毛泽东选集, which combines the official five-volume collection (covering 1925–1957) with the two unauthorized “Jinghuo” 静火 volumes, extending the collection through 1976. Together, these texts comprise Mao’s most significant speeches, essays and directives, establishing the core tenets of Mao Zedong Thought and representing the target style for the familiarization hypothesis (Leese Reference Leese and Cook2014). The second test corpus is the Top 100 Chinese Novels of the 20th Century, a canon compiled in 1999 by Asia Weekly 亚洲周刊 in tandem with a jury of fourteen literary experts (Qiu Reference Qiu1999). The Chinese term xiaoshuo 小说 (lit. “small talk”) is broader than the English “novel,” so this list includes both full-length novels and short story collections from Mainland China, Taiwan and Hong Kong, providing a diverse sample of modern literary styles. All texts were normalized and converted to simplified Chinese. To prevent longer documents from dominating the datasets, a sampling technique was applied to ensure that each document from either corpus could contribute no more than 100 chunks of 1,024 characters (Table 1).

Table 1.

Corpus and model training parameters.

To prevent data leakage, i.e., the inclusion of evaluation content in the training data, an $n$ -gram hashing technique was employed. This is crucial for measuring familiarity with a style rather than memorization of specific passages or even entire documents. All documents in the test corpora were processed to generate a set of unique, character-level 13-grams. This $n$ -gram size is long enough to capture verbatim quotations but short enough to avoid falsely flagging common Chinese expressions; it also aligns with contamination analysis methods used for LLMs (Brown et al. Reference Brown, Mann and Ryder2020; OpenAI 2023). Each 13-gram was hashed using MD5 to create compact fingerprints, which were then used to scan all training documents. Any match was verified with a literal string search, and confirmed documents were discarded from the training set.

The conversion of raw text into numerical identifiers via a tokenizer is a foundational step that shapes the outcome of any computational analysis of language. While state-of-the-art models frequently employ subword tokenizers like byte-pair encoding (BPE), these algorithms introduce significant analytical challenges, particularly for logographic languages like Chinese that lack explicit word delimiters. For example, BPE tokenizers operate by iteratively merging frequent pairs of characters or subwords, which can result in segmentations that are both non-intuitive and inconsistent. A greedy BPE tokenizer might segment 学科技 (“to study electronics”) as 学科 (“discipline”) 技 (“skill”), rather than 学 (“to study”) 科技 (“electronics”). This happens because 学科 (“discipline”), an otherwise valid word, has been learned first by the tokenizer during training and thus appears earlier in the list of merges. Similarly, 第二天 (“the next day”) might be merged into a single token, while 第三天 (“the third day”) might not, as the former occurs more frequently in the training data, etc.

While such inconsistencies may not drastically alter aggregate perplexity scores across a large corpus, they can severely distort the close analysis of individual sequences, mistaking the quirks of a tokenizer for genuine authorial choices. A BPE tokenizer imposes its own statistical “theory” of language onto a text, a theory biased towards common symbol collocations found in its training data. This pre-processing can effectively neutralize the very stylistic effects under study; a sequence of characters might be collapsed into a single token, artificially lowering its perplexity and rendering the author’s stylistic choice invisible to the model. In computational studies of classical Chinese poetry, for instance, the fact that DeepSeek R1 represents ai’ai 曖曖 (“dimly”) as four tokens [923, 247, 923, 247] but yi’yi 依依 (“softly”) as one token [66660] can severely impact character-to-character comparisons (Kurzynski, Xu, and Feng Reference Kurzynski, Xu and Feng2025). To circumvent such inconsistencies, this study treats each Chinese character as a distinct token, allowing for a more fine-grained analysis of which characters in multi-character words and phrases elicit surprise. A custom character-level tokenizer was constructed by training on the first two million documents from the Fineweb-Edu corpus, and the resulting vocabulary was supplemented with single-character tokens from the widely used bert-base-chinese model (Devlin et al. Reference Devlin, Chang and Lee2019). The final vocabulary consists of 20,192 tokens and was used consistently across all experiments in this study.

Experiment 1: Macroscopic analysis of Maospeak

The first experiment is a macroscopic study examining how a language model’s statistical worldview is altered when its linguistic environment becomes saturated with a powerful idiolect. “Maospeak” (Mao yu 毛语 or Mao wenti 毛文体 in its written form) refers to the militant, ideologically charged language style that emerged during the Mao Zedong era (1949–1976) in the People’s Republic of China (Li Reference Li1998). While Mao himself was a creative rhetorician, the institutionalization of his discourse transformed his words into instruments of mass indoctrination, producing a ritualistic culture of near-compulsory recitation (Lifton Reference Lifton1956; Leese Reference Leese2011; Kiely Reference Kiely2014). This engineered form of discourse became a central feature of everyday life, particularly during the Cultural Revolution (1966–1976), when proficiency in Maospeak became essential for survival amid massive outbursts of violence (Ji Reference Ji2004; Juntao and Brady Reference Juntao, Brady and Brady2012; Schoenhals Reference Schoenhals2007). Though Mao passed away in 1976, his distinctive language continues to influence contemporary Chinese speech, from jingoistic discourse to colloquialisms like xiaomie (消灭, lit. “to annihilate”) for finishing a meal or the encouraging phrase haohao xuexi, tiantian xiang shang (好好学习, 天天向上, “study well, every day upwards”) (Link Reference Link2012, Reference Link2013; Barmé Reference Barmé1996).

The experiment tests a dual hypothesis. First, I hypothesize that a dominant political style has a representative phraseology that, through frequent exposure, becomes automatic in the Shklovskian sense. The experiment aims to identify these core expressions by tracking changes in model’s loss, predicting that as the model is exposed to Maospeak, its loss will decrease significantly on ideologically reinforced phrases, specifically those that were either absent from the training data (Base Corpus) and/or frequently repeated in the fine-tuning data (Mao Corpus). Second, I hypothesize that as the model overfits on Maospeak, the stylistic and lexical patterns of literary prose (Novels Corpus) will become increasingly alien. The literary language, in essence, will become more difficult for the model to predict as its perceptual baseline shifts.

The experiment uses a GPT-2 style Transformer model pre-trained from scratch on the Base Corpus for one epoch and then fine-tuned for five epochs on the Selected Works (Table 1). The core analysis compares the model’s performance on the Mao Corpus before fine-tuning (“Pre-trained”) and after (“Epoch 5”). To track how the model internalizes this specific discourse, I deliberately deviate from standard practice and compute loss on the fine-tuning corpus itself. If $Q\left({x}_i|{x}_{<i}\right)$ denotes the probability that model $Q$ assigns to token ${x}_i$ given the preceding tokens, and if, following Shannon, the amount of surprise can be represented as a negative logarithm of the event’s probability (unlikely events are surprising), then for a sequence of $n$ tokens, the cross-entropy loss is the average of all individual surprises:

(1) $$\begin{align}L=-\frac{1}{n}\sum \limits_{i=1}^n\log Q\left({x}_i|{x}_{<i}\right).\end{align}$$

To identify the most significantly learned phrases, loss on each character was calculated across all documents in the Mao Corpus using a sliding window with a 256-character context. Only 16-character phrases (called “patterns” below) where every character was predicted with a full context were considered, eliminating bias from variable context lengths. For each valid pattern, aggregate loss was computed as the arithmetic mean of its character-level losses. The improvement on each pattern was quantified as the difference between the losses of the pre-trained and the fine-tuned model. All discovered patterns were then sorted in descending order of this loss drop.

As Tables 2 and 3 show, the most significant gains in predictive certainty occurred on phrases central to the era’s political machinery. The model learned to anticipate the formulaic language used to categorize class adversaries, from the canonical list of “landlords, rich peasants, counter-revolutionaries, bad elements, and Rightists” (Rank 2) to the ubiquitous label for internal party targets: the “unrepentant capitalist-roader” (Rank 1) and the “small handful of capitalist-roaders in authority” (Rank 12). This familiarization extended to highly specific political targets, including internal factions like the “Gao Gang-Rao Shushi anti-Party alliance” (Rank 4), external ideological foes such as the “Soviet revisionist renegade clique” (Rank 3), and personalized lists of enemies, both domestic (“Zhang Bojun, Luo Longji, Zhang Naiqi,” Rank 9) and international (“Chairman Chiang, Syngman Rhee, and Ngo Dinh Diem,” Rank 11). The model also mastered the rigid, legalistic language of policy, such as the principle that the “chief culprits shall be punished, those who were coerced are not prosecuted, and those who perform meritorious service shall be rewarded” (Rank 16) and the formulation to “resolutely, thoroughly, wholly, and completely annihilate” the enemy (Rank 8). Beyond political labeling, the model also captured the discourse’s distinctive rhetorical features, such as Mao’s characteristic use of classical allusions repurposed for political ends, like the phrase “Sun Wukong drilling into Princess Iron Fan’s belly to make trouble” (Rank 10), which co-opts a well-known episode from the classic Chinese novel Journey to the West 西游记 to describe subversive tactics.

Table 2.

Top 20 most significantly learned 16-character phrases with context.

Table 3.

Top 20 most significantly learned 16-character phrases with context (continued).

These findings illustrate a process of Shklovskian familiarization, where a powerful idiolect turns slogans, labels, rhetorical strategies and even conversational tics into the automatic discursive background, a process which “replaces things with symbols” (Shklovsky Reference Shklovsky and Berlina2017, 79). The new symbolic order functioned as political common sense, establishing the terms of legitimacy and effectively rendering the new regime’s authority self-evident. This principle found its ultimate expression in Lin Biao’s declaration that “one sentence of Chairman Mao’s is worth ten thousand of ours,” creating a high-stakes environment where a mere slip of the tongue could have fatal consequences (Schoenhals Reference Schoenhals1992, 19–20). The complementary phenomenon of defamiliarization can be illustrated by comparing the model’s average perplexity on 16-grams and entire documents in both test corpora as it is progressively fine-tuned on Maospeak. As shown in Figure 1, a large difference exists even for the pre-trained model, suggesting that the stylistic gap between the Novels Corpus and the Base Corpus is wider than that between the Mao Corpus and the Base Corpus. As the model specializes in Mao’s text, the more varied, lexically diverse, and structurally complex language of fiction becomes increasingly alien and harder to predict. This trade-off exemplifies what one might call “cognitive overfitting,” a process where becoming an “expert” in a narrow linguistic domain, without access to external stimuli, leads to a quantifiable loss of fluency in another.

Figure 1.

Comparison of average perplexities on 16-grams (top) and 1,024-token documents (bottom), as calculated on 1,000 trials with 10,000 samples per trial.

Experiment 2: Microscopic analysis

Transitioning from a macroscopic to a microscopic view, the second experiment examines how statistical familiarization manifests at the character level. To this end, I use perplexity, a more interpretable metric calculated by exponentiating the loss defined in Equation (1). A perplexity of 5, for instance, indicates that the model is as uncertain as if it were choosing the next token from five equally likely options. By plotting per-character perplexity for representative passages, we can observe precisely which linguistic elements are initially surprising and how fine-tuning reduces this surprise, effectively “indoctrinating” the model. The figures below illustrate this process, with the yellow line representing the base model’s initial perplexity and the darkening shades of orange and red showing the progressive decrease across five epochs.

All plots exhibit a characteristic “waterfall” pattern: the first character of a key phrase surprises the model, but once this character is revealed, it strongly constrains what can follow, causing subsequent characters to cascade downward with near-certainty. This phenomenon, where the beginning of a sequence is more informative and thus harder to predict than its end, is a general feature of language modeling, whether applied to words (Pimentel, Cotterell, and Roark Reference Pimentel, Cotterell and Roark2021; Pagnoni et al. Reference Pagnoni, Pasunuru, Rodriguez, Che, Nabende and Shutova2025) or chapter breaks (Sun, Thai, and Iyyer Reference Sun, Thai and Iyyer2022). What is distinctive about Maospeak is the length and frequency of such rigid sequences, which severely limit the range of valid continuations.

Bearing resemblance to what Shklovsky described as an automatized perception in which objects are recognized “by their initial features” (Shklovsky Reference Shklovsky and Berlina2017, 79), this characteristic is clearly visible across the three examples (Figure 2). The first passage (top figure) highlights the model’s internalization of characteristic rhetorical structures, featuring the militant formulation 坚决、彻底、干净、全部地消灭掉 (“resolutely, thoroughly, wholly, and completely annihilate,” positions 26–39). The base model is initially very surprised by the sequence of four parallel adverbs. While 坚决 (resolutely) is a common word, the piling on of three more in this escalating sequence is a stylistic hallmark of Maoist rhetoric. The adverbs are followed by an equally characteristic term, niugui sheshen 牛鬼蛇神 (lit. “cow demons and snake spirits”), originally rooted in Buddhist sutras but repurposed by Mao for political agitation and persecution of intellectuals (Lee Reference Lee2014, 49–51). An additional check reveals that this exact rhetorical structure is used widely across the Selected Works, often in nearly identical contexts:

• • 对于任何敢于反抗的反动派, 必须坚决、彻底、干净、全部地歼灭之 (“As for any reactionaries who dare to resist, they must be resolutely, thoroughly, wholly, and completely annihilated”).

• • 在此种情况下, 我们命令你们:奋勇前进, 坚决、彻底、干净、全部地歼灭中国境内一切敢于抵抗的国民党反动派 (“In these circumstances, we order you: advance bravely, and resolutely, thoroughly, wholly, and completely annihilate all Kuomintang reactionaries within China who dare to resist”).

• • 一定要把日本侵略者及其忠实走狗坚决、彻底、干净、全部地消灭掉 (“The Japanese aggressors and their loyal lackeys must be resolutely, thoroughly, wholly, and completely annihilated”).

• • 也罢, 任何这类的欺骗, 必须坚决、彻底、干净、全部地攻击掉, 决不容许保留其一丝一毫 (“So be it; any such deception must be resolutely, thoroughly, wholly, and completely annihilated, with not the slightest trace allowed to remain”).

The second passage (middle figure) showcases the learning of a canonical list of political labels: 地、富、反、坏、右 (“landlords, rich peasants, counter-revolutionaries, bad elements, and Rightists,” positions 33–39). For the base model, this string of single characters is nonsensical and generates extremely high perplexity. Although the characters are common individually, their specific concatenation as an enumeration of class enemies is unique to Maoism.

The third passage (bottom figure) provides a clear example of the model learning specific ideological jargon, containing the phrase 在反对以苏修叛徒集团为中心的现代修正主义的斗争中 (“in the struggle against modern revisionism centered on the Soviet revisionist renegade clique”). The base model (yellow line) displays massive perplexity spikes on the key term 苏修叛徒集团 (“Soviet revisionist renegade clique,” positions 40–44). This is a highly specific piece of Cold War-era verbiage that is rare in general-purpose text and thus highly surprising. Like in the previous example, with each epoch of fine-tuning, the perplexity on this exact phrase plummets as the model learns to recognize it as a core component of the discourse (Figure 2).

Figure 2.

Per-character perplexity plots for three excerpts from the Selected Works of Mao Zedong. The yellow line (Base) shows the perplexity of the pre-trained model, while the lines from orange to red show the decreasing perplexity over five epochs of fine-tuning on the Mao Corpus. The $x$ -axis displays the Chinese character sequence, and the $y$ -axis shows perplexity on a logarithmic scale.

As Ji Fengyuan argues, linguistic engineering was central to Maospeak as a tool of political control (Ji Reference Ji2004). Aligning with a weak interpretation of the Sapir–Whorf hypothesis, she suggests that language influences thought by making certain cognitive pathways more accessible. The constant repetition of slogans and labels activated associative networks and cognitive schemas tied to concepts like “class struggle” and “proletarian revolution,” aligning public sentiment with ideological goals and providing ready-made justifications for denunciation and violence, effectively automating cruelty. However, Ji also cautions against an Orwellian view of language as a totalizing instrument of domination (Ji Reference Ji2004, 134–143). The low-perplexity canyons are interspersed with less predictable tokens, revealing moments of instability; some grammatical connectives or less common phrasings elicit moderate surprise even after fine-tuning, representing points where Maospeak gives way to the more varied patterns of general language. Conversely, common function words like 的 (of), 在 (at) and 中 (in) consistently show low perplexity from the outset. The initial perplexity spikes are therefore reserved for elements stylistically and ideologically unique to Maospeak, underscoring the method’s ability to isolate the process of familiarization with the specific features of a discourse against the backdrop of general linguistic competence.

Experiment 3: $N$ -gram entropy

The previous experiments suggested that Maospeak operates by relying on a constrained set of high-frequency, low-perplexity phrases, making it more statistically compressible than other forms of discourse. To test this hypothesis quantitatively, I conducted an $n$ -gram entropy analysis. The experiment compared the Mao Corpus against the Novels Corpus by repeatedly sampling 100,000 $n$ -grams (from $n=2$ to $n=16$ ) from each and calculating the Shannon entropy. As a measure of predictability, lower entropy signifies a more repetitive text (approaching zero for a text with no variation), while higher entropy indicates greater linguistic variety (maximized in a uniform distribution of $n$ -grams).

The results (Figure 3) support the initial hypothesis. At every $n$ -gram size tested, the Mao Corpus exhibited lower entropy than the literary corpus, indicating a more constrained and formulaic linguistic system. The difference was most pronounced for short phrases; for bigrams ( $n=2$ ), the entropy of the Mao Corpus was 13.442 bits compared to 14.331 for the novels. This entropy gap narrowed as $n$ -gram size increased, which is expected: as $n$ grows, $n$ -grams become so specific that they are unlikely to repeat within a fixed sample, causing the entropy of both corpora to approach the theoretical maximum for a sample of this size ( ${\log}_2\left(\mathrm{100,000}\right)\approx 16.6$ bits). The key finding is that the Mao Corpus remains measurably less entropic at smaller $n$ -gram sizes, confirming that its repetitiveness is most evident in its foundational vocabulary and short collocations. The heavy repetition of specific $n$ -grams, resulting in lower entropy, corresponds to the “waterfall” patterns observed in the fine-tuned model’s perplexity scores.

Figure 3.

Shannon entropy comparison between Mao’s Selected Works and Chinese novels across different $n$ -gram sizes on 1,000 trials.

Experiment 4: Impact on literature

Finally, I turn to works of literature to examine how the opposing impulses of familiarization and defamiliarization manifest in novels. By examining how three distinct modern Chinese authors engage with, repurpose, or resist predictable linguistic patterns, we can observe how literary art creates meaning through surprise. While familiarization reduces perplexity through repeated exposure, defamiliarization emerges (indirectly) as a by-product of overfitting and (directly) as an aesthetic intervention. Therefore, to explore this phenomenon, the following passages were selected subjectively for their aesthetic and stylistic significance.

The first passage (Figure 4, Top) comes from a pivotal scene in Chapter 24 of Zhang Wei’s 张炜 (b. 1956) novel The Ancient Ship 古船 (1987). The story is set in Wali, an imagined microcosm of Chinese society which witnesses the vicissitudes of the communist revolution. In the passage, a “red-faced young man,” representing the fervent revolutionary youth, delivers an incendiary speech at a public struggle session, aiming to mobilize the crowd to overthrow the town’s authorities.

Figure 4.

Per-character perplexity plots for three excerpts from novels. The light blue line (Base) shows the perplexity of the pre-trained model, while the darker lines show the decreasing perplexity over five epochs of fine-tuning on the Mao Corpus. The $x$ -axis displays the Chinese character sequence, and the $y$ -axis shows perplexity on a logarithmic scale. Top: Zhang Wei’s The Ancient Ship; Middle: Lilian Lee’s Farewell My Concubine; Bottom: Dung Kai-cheung’s Works and Creations.

Zhang Wei immerses the reader in the linguistic reality of the turbulent era that he himself experienced as a child. The speech is a barrage of Maoist-era slogans showcasing the romanticization of violence (“spill our blood”) and rhythmic chants (“Fight! Fight!”). The most potent phrase is the proverb, “Those who are not afraid of being cut to a thousand pieces dare to pull the emperor off his horse” (positions 29–41), which comes from another Chinese classic, the Dream of the Red Chamber, and which gained new life through Mao’s use. By placing this slogan in the young man’s mouth, Zhang Wei not only highlights the central paradox of the era: the nation’s supreme leader encouraging violent rebellion against all “emperors,” but also masterfully imitates the dominant language style of the period. The Maoist slogan is the only part of the sequence that experienced a significant drop in perplexity (Figure 4, Top). As such, the method was able to identify the Maospeak embedded within the novel.

The second case study examines a passage from Farewell My Concubine 霸王别姬, the seminal 1985 novel by Hong Kong author Lilian Lee 李碧华 (b. 1958). The novel chronicles the tumultuous relationship between two Peking opera performers, Duan Xiaolou and Cheng Dieyi, against the backdrop of China’s political upheavals. The passage in question appears early in the novel, as the protagonists, then young boys, endure the brutal training of an opera troupe. After Xiaolou is caught secretly helping Dieyi ease the pain of a torturous leg-stretching exercise, Master Guan punishes the entire troupe with collective beating. The passage offers a piercing narrative insight into the master’s psychology and the constrained nature of masculinity that defines the novel’s world.

The perplexity distribution for this passage (Figure 4, Middle) shows how literary language creates its effects through moments of surprise. The highest perplexity spikes occur on the creative juxtapositions: 凶狠地打哈欠 (“a ferocious yawn,” 45–51) or 向… 弱小吼叫 (“to roar at the weak,” 59–62). These phrases elicit high perplexity likely because figurative language is less probable than literal phrases (Pedinotti et al. Reference Pedinotti, Di Palma, Cerini, Bastings, Belinkov and Dupoux2021). Even more telling is the passage’s use of pre-modern and classical Chinese phrasings, which become increasingly unfamiliar to the model as it is fine-tuned on Maospeak. For instance, the adjective 堂堂 in 堂堂男子 (“dignified man”) is a formal, literary term for “imposing” or “grand” that ironically evokes here a traditional sense of masculinity, a register alien to the functional dialect of Maospeak. Likewise, the final question, 机会安在 (lit. “the chance is where?”), uses the character 安 in its classical function as an interrogative pronoun (“where”), a usage largely replaced by words like 哪里 in modern vernacular and mostly absent from the Mao Corpus. The fine-tuned model finds both of these literary phrasings even more anomalous than it did initially, causing their perplexity to increase.

These high-entropy word choices, which slow down the narrative, are set against a backdrop of linguistic convention (Bizzoni, Feldkamp, and Nielbo Reference Bizzoni, Feldkamp and Nielbo2024) and narrative scaffolding that glues together the surprising tokens. The passage opens with just such a phrase: 不如意的人太多了 (“too many people whose lives are not as they wish”). This is a low-perplexity, almost proverbial, statement that eases the reader into the scene by establishing a familiar emotional context. This kind of predictable framing anchors the unique description of events in familiar literary conventions, signaling the narrative’s fictional nature and making the subsequent, high-perplexity surprises more effective.

The final case study focuses on Hong Kong writer Dung Kai-cheung 董启章 (b. 1967), known for his postmodern narratives merging memory, technology, and urban space. A distinctive feature of Dung’s style is his interspersion of standard written Chinese with Cantonese, the non-normative language spoken in Guangdong, Hong Kong, and numerous overseas Chinese communities in Southeast Asia and North America. His novel Works and Creations: As Vivid As Real 天工开物·栩栩如真 (2005) interweaves fictional and autobiographical storylines. In the autobiographical part, Dung’s grandmother, Lung Gam Juk, confides in her husband a painful memory of separation from her brother, which is when she first realized her extraordinary ability to hear radio waves:

This seemingly straightforward scene blends free indirect speech, unmarked direct quotations, and internal monologue. The perplexity plot (Figure 4, Bottom) shows how Cantonese terms like 嗰 (“that”), 系 (“to be”) and 既 (“of”) disrupt the perplexity arc. The surprise is enhanced because the Cantonese statement is not announced with quotation marks, making an abrupt switch from the narrator’s to the character’s voice. Meanwhile, novel-specific terms like “upright” (正直人) and character names challenge the model’s predictive capacities, providing a clear example of what empirical literary studies call “foregrounding” (Peer et al. Reference Peer, Sopčák, Castiglione, Kuiken and Jacobs2021). As such, the heteroglossic novel “puts the brakes on perception” (Shklovsky Reference Shklovsky and Berlina2017, 94), challenging both human and machine readers. Written Cantonese contains many characters that are absent from standard written Chinese but are essential to transcribe words used by Cantonese speakers on a daily basis (Snow Reference Snow2004). Many Hong Kong writers use these non-standard characters to signal the cultural distinctiveness of their work, and some go as far as to write entire novels in Cantonese. On a technical level, Cantonese expressions resist prediction because they are underrepresented in training data, highlighting the normative assumptions underlying most Chinese LLMs (Xiang et al. Reference Xiang, Chersoni and Li2025; Jiang et al. Reference Jiang, Chen, Chen, Chiruzzo, Ritter and Wang2025).

This use of a topolect as a defamiliarizing device is a classic literary strategy. For language to be poetic, Shklovsky argued after Aristotle, it ought to have “the character of the foreign” (Shklovsky Reference Shklovsky and Berlina2017, 94). He observed that when a literary establishment exhausts the standard language, it turns to regional vernaculars, and so “folk language and literary language have changed places” (Shklovsky Reference Shklovsky and Berlina2017, 95). Although Cantonese can boast significantly richer cultural heritage than “modern standard Chinese,” Dung’s insertion of Cantonese speech operates through similar mechanisms. By deliberately disrupting the automatized perception of written Chinese, the author forces readers—or at least those accustomed only to the standard tongue—to encounter the cultural specificity of his text.

Limitations

Several limitations of this study should be acknowledged. First, the results are contingent on the chosen GPT-2 style architecture and the Fineweb-Edu corpus, which serves as the “neutral” baseline of modern Chinese. A different, perhaps larger, model or a baseline corpus with a different composition (e.g., one richer in classical literature or web-based text) would inevitably alter the specific perplexity values and could shift the perceived boundary between the familiar and the unfamiliar. Furthermore, the fixed context window of 256 tokens imposes an artificial boundary on the model’s predictive horizon, which cannot fully replicate the long-term memory and rich contextual understanding of a human reader. The very fact that each literary excerpt in this study required a brief prose introduction highlights this gap. While there is no scholarly consensus on the optimal context length for such comparisons (Fang et al. Reference Fang, Wang and Liu2024), and internal tests with different window sizes confirmed the overall trends and identified patterns, this parameter remains a significant variable.

Second, it bears repeating that the study’s core analogy, which equates perplexity with cognitive surprise, is a deliberate simplification. Human reading involves embodied cognition, emotional resonance and multi-modal knowledge that the LLMs employed in this study lack, just as fine-tuning only approximates the socio-historical saturation of Maospeak. New model architectures and human-centered artificial intelligence might reduce these gaps, allowing scholars to better model historical contexts and circumstances.

Finally, the selected hyperparameters, while supported by local testing, necessarily impact the final results. Notably the fine-tuning learning rate and the gradient norm clipping influence the learning process and, if not carefully adjusted, might lead to “catastrophic forgetting” of the base model. An alternative setting would be to progressively contaminate the training dataset and train a series of different models, thus bypassing the fine-tuning stage. This approach is less practical than the one adopted in this study, however, as fine-tuning is less computationally expensive than training entire models from scratch.