Feature-based detectors

Classical machine learning methods are widely employed for detecting generated texts. Linear classifiers over TF-IDF character, sub-word, and word N-grams can serve as lightweight and strong baseline detectors. (Manjavacas et al. Reference Manjavacas, De Gussem, Daelemans and Kestemont2017; Solaiman et al. Reference Solaiman, Brundage, Clark, Askell, Herbert-Voss, Wu, Radford, Krueger, Kim, Kreps, McCain, Newhouse, Blazakis, McGuffie and Wang2019; Ippolito et al. Reference Ippolito, Duckworth, Callison-Burch and Eck2020).Badaskar et al. (Reference Badaskar, Agarwal and Arora2008) train detectors over morphological, syntactic, and discourse features, such as POS tags, syntax-based LM’s grammaticality scores, and coherence. Another group of features is based on stylometry, a branch of computational linguistics that relies on statistical methods for authorship attribution and analysis of literary style (Holmes, Reference Holmes1994, Abbasi and Chen, Reference Abbasi and Chen2008; Abbasi and Chen, Reference Abbasi and Chen2008). Stylometric features are used to train detectors and characterise properties of machine-generated and human-written texts (Uchendu et al. Reference Uchendu, Le, Shu and Lee2020, Reference Uchendu, Ma, Le, Zhang and Lee2021). Specific types of stylometric features can capture issues related to TGMs (Fröhling & Zubiaga, Reference Fröhling and Zubiaga2021): (i) lack of syntactic and lexical diversity (POS tags, named entities, and coreference chains), (ii) repetitiveness (N-gram overlap of words and POS tags and counters of stopwords and unique words), (iii) lack of coherence (the appearance and grammatical roles of named entities), and (iv) lack of purpose (lexical features that represent spatial properties, sentiment, opinion, and logic).

The feature-based detectors are interpretable, cost-effective, and helpful when the dataset size is small (Uchendu, Le and Lee Reference Uchendu, Le and Lee2023). Stylometric detectors show usefulness in recognising texts generated with certain decoding strategies (Fröhling & Zubiaga, Reference Fröhling and Zubiaga2021) but are significantly inferior in performance compared to Transformer-based detectors (Schuster et al. Reference Schuster, Schuster, Shah and Barzilay2020; Diwan et al. Reference Diwan, Chakraborty and Shafiq2021; Jones Nurse and Li Reference Jones, Nurse and Li2022).

Transformer-based detectors

The current state-of-the-art approach is fine-tuning a pretrained Transformer LM for the ATD classification task. Zellers et al. (Reference Zellers, Holtzman, Rashkin, Bisk, Farhadi, Roesner and Choi2019) train a linear layer over the hidden representations from the GROVER and GPT-2 TGMs. RoBERTa (Liu et al. Reference Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov2019) has demonstrated an outstanding performance with respect to many TGMs’ configurations and domains (Adelani et al. Reference Adelani, Mai, Fang, Nguyen, Yamagishi and Echizen2020; Fagni et al. Reference Fagni, Falchi, Gambini, Martella and Tesconi2021). The TGM-based detectors identify texts generated by the previous TGMs’ versions better than by the more recent ones, which confirms that newer TGMs generate more human-like texts. Kushnareva et al. (Reference Kushnareva, Cherniavskii, Mikhailov, Artemova, Barannikov, Bernstein, Piontkovskaya, Piontkovski and Burnaev2021) introduce the topological data analysis (TDA) method for ATD, which combines properties of the feature-based and Transformer-based detectors. The TDA features are extracted from the BERT attention matrices that are represented as weighted attention graphs and used to train a linear detector. The features include standard graph properties, descriptive characteristics of barcodes, and distances to attention patterns (Clark et al. Reference Clark, Khandelwal, Levy and Manning2019). The TDA-based detectors outperform TF-IDF-based and neural detectors and show better robustness toward the size of unseen GPT-style TGMs. The Transformer-based detectors are highly effective in ATD tasks and generalisable to out-of-domain sets but computationally expensive.

Zero-shot detectors

An alternative set of methods involves using probability-based measures along with predetermined thresholds (Solaiman et al. Reference Solaiman, Brundage, Clark, Askell, Herbert-Voss, Wu, Radford, Krueger, Kim, Kreps, McCain, Newhouse, Blazakis, McGuffie and Wang2019; Mitchell et al. Reference Mitchell, Lee, Khazatsky, Manning and Finn2023). These methods enable the human-in-the-loop approach, in which a user can recognise whether a text is machine-generated with the assistance of pretrained LMs. The GLTR tool (Gehrmann et al. Reference Gehrmann, Strobelt and Rush2019) facilitates interaction between humans and models by presenting statistical characteristics of a text inferred by the model, thereby enhancing the ability of humans to identify artificial texts. MAUVE (Pillutla et al. Reference Pillutla, Swayamdipta, Zellers, Thickstun, Welleck, Choi and Harchaoui2021) is a statistical detector that determines the difference in distribution between human and neural texts by utilising the KL-divergence. When MAUVE highlights differences between human and neural texts, human identification of texts generated by GROVER and GPT-2 strongly improves. Dugan et al. (Reference Dugan, Ippolito, Kirubarajan and Callison-Burch2020) propose RoFT (Real or Fake Text), a tool that assists in distinguishing between human-written and machine-generated texts. Their work highlights that TGMs have the ability to deceive humans with just a few sentences. Gallé et al. (Reference Gallé, Rozen, Kruszewski and Elsahar2021) explore unsupervised ATD, employing repeated higher-order N-grams. Their findings indicate that certain well-formed phrases occur more frequently in machine-generated texts compared to those written by humans. Although zero-shot detectors tend to underperform compared to the other established methods, they are beneficial in aiding humans to detect machine-generated texts.

Number of texts

Tables 2 and 3 summarise the distribution of texts in CoAT by natural language task, model, and domain. The number of human-written and machine-generated texts is balanced within each natural language task and domain. Texts from the Russian National Corpus are the most common in CoAT (20%). News articles make up a percentage of 19.6% of the total number of texts, followed by strategic documents (18.5%) and texts from Wikipedia (17.5%). Digitalised personal diaries comprise 10%, while texts from social media and machine translation datasets account for 7% and 6%, respectively.

Table 2. General statistics of CoAT by natural language generation task and model

Table 3. Number of texts in each domain and task-specific dataset in CoAT

Length and frequency

Table 4 presents a statistical analysis of CoAT based on the frequency and lexical richness metrics. We compute the token frequency in each text as the number of frequently used tokens (i.e., the number of instances per million in the Russian National Corpus is higher than one) divided by the total number of tokens in the text. The average text length in razdel tokens is similar for the sentence-level natural language generation tasks (back-translation, machine translation, and paraphrase generation), while texts produced by text simplification and summarisation models are of $20.59$ and $31.45$ tokens on average, respectively. The overall average text length is $49.64$ . The distribution of high-frequency tokens in human-written and machine-generated texts is similar within each generation task, comprising of 88% high-frequency tokens on average.

Table 4. Lexical richness metrics by natural language generation task in CoAT. Notations: %=the average fraction of high-frequency tokens. H=human-written texts. M=machine-generated texts

Lexical richness

We evaluate the lexical richness of the CoAT texts using three measures from the lexicalrichness Footnote ^r library: word count, terms count, and corrected type-token ratio (CTTR). The corrected type-token ratio is calculated as $t/\sqrt{2 * w}$ , where $w$ is the total number of all words, including functional, and $t$ is the number of unique terms in the vocabulary. We observe that the ratio of the measures between the natural and artificial texts depends on the task. In contrast to other natural language generation tasks, neural texts generated using the open-ended generation approach are generally longer than the human-written ones, and receive higher richness metrics. The reason is that the models try to generate texts up to the maximum number of 500 tokens, and may produce non-existent words, degenerated textual segments, or rare words.

Stylometric features

Stylometry helps characterise the author’s style based on various linguistic features, which are commonly applied in the ATD and human and neural AA tasks (He et al. Reference He and Rasheed2004; Lagutina et al. Reference Lagutina, Lagutina, Boychuk, Vorontsova, Shliakhtina, Belyaeva, Paramonov and Demidov2019; Uchendu et al. Reference Uchendu, Cao, Wang, Luo and Lee2019, Reference Uchendu, Le, Shu and Lee2020). This section aims to analyse the stylometric properties of human-written and machine-generated texts.

Following the motivation by Fröhling and Zubiaga (Reference Fröhling and Zubiaga2021), we manually inspect a subset of the artificial texts to define stylometric features that potentially capture text generation errors (see Table 5). The features can be informally categorised as follows:

1. (1) Surface and frequency features: (i) the text’s length in characters (Length), (ii) the fraction of punctuation marks in the text (PUNCT), (iii) the fraction of Cyrillic symbols in the text (Cyrillic), and (iv) IPM.

2. (2) Count-based morphological features: the fraction of (i) prepositions (PREP) and (ii) conjunctions in the text. The features are computed with pymorphy2 (Korobov Reference Korobov2015), a morphological analyser for Russian and Ukrainian.

3. (3) Readability measures: (i) Flesch-Kincaid Grade (FKG), (ii) Flesch Reading-Ease (FRE), and (iii) LIX adapted to the Russian language. We use the ruTS library (Shkarin Reference Shkarin2023) to compute the features.

Table 5. Manually picked examples of text generation errors from the train set. The examples are automatically translated into English for illustration purposes

Statistical analysis

Table 6 presents the summary of the stylometric feature values for the human-written and machine-generated texts and each text generator independently.Footnote ^s The Length is attributed to the natural language generation task; texts produced by MT and paraphrase generation models will be naturally shorter. In contrast, ruGPT3-Small, ruGPT3-Medium, and ruGPT3-Large generate longer texts, which is the case for the zero-shot generation setup. At the same time, Cyrillic varies between the models and can be specific to text translations, decoding confusions, copying parts of the input in another language, and hallucinations. We observe that the percentage of high-frequent words (IPM) in the texts is similar among the generators. The morphological and readability features differ on average between the TGMs and humans and between the TGMs. The results are supported by the Mann-Whitney U test used to compare the differences between the distributions of the stylometric features in human-written and machine-generated texts. There is a significant difference between the mean values for all features except for IPM and PREP, with $\alpha =0.05$ .

Table 6. The average values of the stylometric features in the corpus subset. Machine refers to the machine-generated texts

Analysis of the feature space

We use the principal component analysis (PCA; Pearson Reference Pearson1901) on the stylometric features computed on the weighted corpus subset. Figure 1 illustrates the 2-dimensional distribution of the features by generative model. A large overlapped portion among the texts indicates that the stylometric features may not be useful in solving the ATD tasks.

Figure 1. 2-dimensional distribution of the corpus subset using PCA.

Figure 2. The distribution of the FRE (left) and Cyrillic (right) features between the human-written and machine-generated texts in the corpus subset. The difference between the mean values is statistically significant according to the Mann-Whitney U test, with the p-values equal to 0.002 and 0.0, respectively.

Discussion

The linguistic analysis indicates that distributions of most of the considered stylometric features underlying the human-written and machine-generated texts are not the same (see Figure 2). However, a substantial portion of the texts overlaps, meaning that the properties of the texts are similar among the generators. We conclude, that due to this overlap, it might be challenging to distinguish between the human-written and machine-generated texts and identify the author of a given text by utilising only stylometric features.

Non-neural detectors

We use two non-neural text detectors via the scikit-learn library (Pedregosa et al. Reference Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss and Dubourg2011): a data-agnostic classifier, referred to as “Majority,” that predicts the most frequent class label in the training data, and a logistic regression classifier trained on TF-IDF features computed on word N-grams with the N-gram range $\in [1; 3]$ , denoted as “Linear”.

Transformer-based detectors

We experiment with four monolingual and cross-lingual Transformer LMs: ruBERT-base (Zmitrovich et al. Reference Zmitrovich, Abramov, Kalmykov, Kadulin, Tikhonova, Taktasheva, Astafurov, Baushenko, Snegirev, Shavrina, Markov, Mikhailov and Fenogenova2024; 178M), ruRoBERTa-large (Zmitrovich et al. Reference Zmitrovich, Abramov, Kalmykov, Kadulin, Tikhonova, Taktasheva, Astafurov, Baushenko, Snegirev, Shavrina, Markov, Mikhailov and Fenogenova2024; 355M), XLM-R-base (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer and Stoyanov2020; 278M), and RemBERT (Chung et al. Reference Chung, Fevry, Tsai, Johnson and Ruder2020; 575M).

Performance metrics

The accuracy score is used as the target metric in the binary classification problems (Sections 5.2; 5.3; and 5.5), and macro-averaged $F_1$ is used as the target metric in the multi-class classification problem (Section 5.4). The results are averaged over three restarts with different random seeds.

Training details

We maximise the validation set performance by running a grid search over a set of hyperparameters. We tune the $L_2$ regularisation coefficient $C \in \{0.01, 0.1, 1.0\}$ to train the logistic regression model. We use the Transformer LMs’ weights and codebase for fine-tuning and evaluation from the HuggingFace Transformers library (Wolf et al. Reference Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu, Le Scao, Gugger, Drame, Lhoest and Rush2020). The detectors are fine-tuned for $5$ epochs over the learning rate $10^{-5}$ , with a fixed weight decay of $10^{-4}$ and batch sizes of $32$ for RemBERT and $64$ for the other LMs.

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion (172k/24k/49k examples). We use the sets to create train, validation, and private test subsets for each experiment described below: (i) artificial text detection (Section 5.2), (ii) artificial text detection by natural language generation task (Section 5.3, (iii) authorship attribution (Section 5.4), (iv) robustness towards unseen TGM (Section 5.5.1), and (v) robustness towards unseen text domain (Section 5.5.2).

In Sections 5.2, 5.3, and 5.5, we train or fine-tune binary classifiers to distinguish between text written by a human and text generated with a TGM. In Section 5.4, the machine-generated label in the artificial text detection task is broken into 13 model names in the authorship attribution task. In each experiment configuration, we use the corresponding subsets of the CoAT train, validation, and private test set splits, which are balanced by the number of examples per target class, natural language generation task, TGM, and text domain.

Task formulation

This experiment aims to evaluate the performance of detectors in determining if a given text is automatically generated or written by a human. This task is framed as a binary classification problem with two labels: H (human-written) and M (machine-generated).

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion (172k/24k/49k examples).

Figure 3. The web interface used for human evaluation on the artificial text detection task.

Human baseline

We conduct a human evaluation on the artificial text detection task using a stratified subset of 5k samples from the test set. The evaluation is run via Toloka (Pavlichenko, Stelmakh, and Ustalov Reference Pavlichenko, Stelmakh and Ustalov2021), a crowd-sourcing platform for data labelling.Footnote ^t The annotation setup follows the conventional crowd-sourcing guidelines for the ATD task and accounts for methodological limitations discussed in Ippolito et al. (Reference Ippolito, Duckworth, Callison-Burch and Eck2020); Clark et al. (Reference Clark, August, Serrano, Haduong, Gururangan and Smith2021). We provide a full annotation instruction in Appendix A and an example of the Toloka web interface in Figure 3.

The human evaluation project consists of an unpaid training stage and the main annotation stage with honeypot task for tracking annotators’ performance. The honeypot tasks are manually picked quality verification examples from the CoAT training set, which are mixed in with the main unlabelled examples. We compute the annotation performance by comparing the annotators’ votes with the honeypot examples’ gold labels. Before starting, the annotator receives a detailed instruction describing the task and showing annotation examples. The instruction is available anytime during the training and main annotation stages. Access to the project is granted to annotators ranked top- $70$ % according to the Toloka rating system. Each annotator must finish the training stage by answering at least 27 out of 32 examples correctly. Each of the trained annotators gets paid. The pay rate is on average $2.55/h, which is twice the amount of the hourly minimum wage in Russia. We aggregate the majority vote labels via the dynamic overlap from three to five trained annotators after (i) discarding votes from annotators whose annotation performance on the honeypot tasks is less than 50% and (ii) filtering out submissions with less than 15 s of response time per five texts.

Results

Table 7 presents the results of this experiment. Transformer-based detectors outperform non-neural detectors by a wide margin with an average accuracy of 0.864. The highest accuracy scores were achieved using ruRoBERTa, followed by RemBERT, ruBERT, and XLM-RoBERTa. Notably, using monolingual ruRoBERTa and ruBERT led to a significant improvement in accuracy scores, despite having fewer parameters than XLM-RoBERTa and RemBERT. At the same time, human annotators significantly underperform the detectors. The human evaluation yields an overall accuracy of 0.66, which is 0.07 points lower than that of non-neural detectors and 0.2 points lower than that of neural detectors.

Table 7. Accuracy scores of the detectors on the artificial text detection task

Figure 4. Macro-F1 scores of the detectors on the artificial text detection task grouped by five quintiles of the text length.

Effect of length

We divide the test set into five groups of equal size based on the text length to examine the effect of text length on the performance. As depicted in Figure 4, the F1-macro-scores of learnable detectors improved monotonically as the text length increases. For the shortest texts, non-neural and neural detectors achieved F1-macro-scores of 0.6 and 0.75, respectively. The scores increase significantly, resulting in near-perfect predictions of 0.95 to 0.99 for the longest texts. The differences between the four Transformer-based detectors are most prominent in shorter texts but became less significant as the text length increases.

Humans exhibit comparable or better performance than non-neural detectors when it comes to texts that are up to 31 words. However, humans seem to face difficulty in classifying texts that fall within the range of 32–92 words, while performing best in identifying the longest texts. A possible explanation for this consistent performance of humans within the 0.6 to 0.75 range of the macro-F1 could be that humans tend to rely more on surface-level features, which may have a similar distribution across all text length groups.

Discussion

The results demonstrate that state-of-the-art Transformer-based models can be relatively successful in distinguishing human-written texts from machine-generated ones for the Russian language. However, one can quickly notice a rather stark contrast between the best scores obtained on the CoAT test set in binary setup (0.86 accuracy) and scores obtained for a similar setup in English (0.970 accuracy; see Uchendu et al. (Reference Uchendu, Le, Shu and Lee2020) for reference). We attribute this disparity not to the language discrepancy but primarily to the nature of the texts. In the English setup, the average text length was 432 words, which is nearly nine times longer than the CoAT’s 50-word average. Our findings align with the works by Munir et al. (Reference Munir, Batool, Shafiq, Srinivasan and Zaffar2021); Stiff and Johansson (Reference Stiff and Johansson2022), which report that shorter texts are the most challenging for both human annotators and computational methods.

The low results in human evaluation are consistent with recent works by Karpinska et al. (Reference Karpinska, Akoury and Iyyer2021); Uchendu et al. (Reference Uchendu, Ma, Le, Zhang and Lee2021), which underpin the difficulty of the generated text detection task for crowd-sourcing annotators. These works advise hiring experts trained to evaluate written texts or conduct multiple crowd-sourcing evaluation setups with extensive training stages.

Task formulation

In this experiment, we create six datasets with respect to each natural language generation task. The detectors are independently trained and evaluated on the human-written and machine-generated texts from the same task. This setup allows for estimating the complexity of the tasks. The lower the performance scores, the more natural the TGMs in a particular task.

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion: 9.8k/1.4k/2.7k (machine translation), 12.4k/1.7k/3.4k (back-translation), 31.8k/4.4k/8.9k (paraphrase generation), 26.6k/3.7k/7.3k (text summarisation), 28k/3.8k/7.9k (text simplification), and 63k/9.2k/17.9k (open-ended generation).

Results

The results of our experiment are presented in Table 8. Averaged macro- $F_1$ scores are presented in Table 9. We observe that the detectors’ performance rankings are consistent with their performance in the artificial text detection setup (Section 5.2). ruRoBERTa consistently outperforms the other detectors with an average macro- $F_1$ of 0.814, while other models show moderate performance. Analysing the results by a natural language generation task, we find that machine-translated texts are more difficult to detect, where the best detector performs only slightly better than random prediction, with an accuracy of 0.621 and macro- $F_1$ of 0.612. Tasks such as Back-translation, Paraphrase Generation, and Text Simplification are of intermediate difficulty, resulting in an accuracy of 0.768 and macro- $F_1$ of 0.765 or above. Finally, we find that Text Summarisation and Open-ended Generation are much easier to detect. In fact, the detectors made near-perfect predictions in Open-ended Generation, with accuracies and macro- $F_1$ scores reaching up to 0.99.

Table 8. Macro- $F_1$ and accuracy scores by a natural language generation task

Table 9. Averaged macro- $F_1$ by a natural language generation task

Discussion

We suggest that the diversity of TGMs’ output and the degree of control over the outputs in the task affect the task difficulty. The TGM outputs of Machine Translation and Paraphrase Generation are semantically constrained by the inputs, thus it is less likely for these TGM to produce in-plausible or repetitive outputs. Open-ended Generation models suffer from hallucination and repetitions, which at scale can be learnt by detectors. At the same time.

Task formulation

The author attribution task aims at determining the author of a given text. The task is framed as a multi-class classification problem with 14 target classes: a human author and 13 TGMs. In this experiment, we use the same dataset as in Section 5.2, but this time instead of binary labels we use the source TGM’s labels as prediction targets. In this setup, the dataset is imbalanced: 50% of samples are human-written and the other 50% of samples contain outputs of 13 TGMs. In this case, we rely on macro- $F_1$ as the main performance metric.

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion (172k/24k/49k examples).

Results

Our findings, shown in Table 10, demonstrate that RuRoBERTa significantly outperforms other models with a macro- $F_1$ score of 0.521. RemBERT and ruBERT performed similarly with macro- $F_1$ scores of 0.496 and 0.476, while XLM-RoBERTa achieved a lower macro- $F_1$ score of 0.451. These findings align with the results of artificial text detection (Section 5.2) and artificial text detection by task (Section 5.3). In terms of TGMs, the neural detectors achieved the highest average performance of 0.75 $F_1$ in detecting ruT5-Base, while the lowest average performance of 0.112 $F_1$ was observed in detecting ruT5-Large. However, the TGM size did not significantly affect the performance of neural detectors in the ruGPT3 family, as the difference in $F_1$ between detecting ruGPT3-Small and ruGPT3-Large was only 0.01. In summary, neural detectors exhibit significantly higher accuracy in identifying human-written texts (with an average $F_1$ score of 0.866) compared to determining the source TGM.

Table 10. $F_1$ scores of the authorship attribution task by the target TGM

Discussion

Authorship attribution is important when legal requirements demand revealing text generation models for transparency, claiming intellectual property, or replicating text generation. The task of authorship attribution is more challenging than the binary artificial text detection task, as observed in recent works (Uchendu et al. Reference Uchendu, Le, Shu and Lee2020, Reference Uchendu, Ma, Le, Zhang and Lee2021). The low-performance scores indicate that it is challenging to differentiate between the TGMs, as they may lack distinct features that set them apart from each other, which might align with the results of the corpus linguistic analysis in Section 4.

Task formulation

This experiment setting tests the detectors’ robustness towards the size of unseen GPT-based TGMs. The detectors are trained or fine-tuned on the human-written texts and texts generated by one of the GPT-based models (ruGPT3-Small, ruGPT3-Medium, and ruGPT3-Large) as described in Section 5.1 and further inferred to detect texts from the out-of-domain GPT-based TGMs, i.e., those that are not present the train set. Consider an example for the ruGPT3-Small model, where we train the detectors on a mixture of human-written texts and texts generated by only ruGPT3-Small, and evaluate the detectors on three mixtures of human-written texts and texts generated by ruGPT3-Small, ruGPT3-Medium, and ruGPT3-Large.

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion: 26.2k/3.7k/7.7k (ruGPT3-small), 26k/3.7k/7.6k (ruGPT3-medium), and 28.4k/4k/8.2k (ruGPT3-large).

Results

The results are presented in Figure 5. We observe that larger models’ outputs are more challenging to identify as all figures show an evident drop in performance. However, training on larger models’ outputs improves the detectors’ robustness, as lines in the left-most plot become more flattened. Analysing the results by the detector, we find that all neural detectors achieve a strong performance of more than 90% accuracy. ruRoBERTa performs on par with RemBERT while having fewer parameters, and the linear detector receives moderate performance falling behind the Transformers by a large margin.

Discussion

Overall, the results align with the works of Solaiman et al. (Reference Solaiman, Brundage, Clark, Askell, Herbert-Voss, Wu, Radford, Krueger, Kim, Kreps, McCain, Newhouse, Blazakis, McGuffie and Wang2019) and Kushnareva et al. (Reference Kushnareva, Cherniavskii, Mikhailov, Artemova, Barannikov, Bernstein, Piontkovskaya, Piontkovski and Burnaev2021). The ATD task can become more challenging with scaled TGMs since the TGMs’ size directly determines the detector performance. However, the detectors receive the most optimal performance concerning different TGMs’ sizes when trained on outputs from the largest ones.

Task formulation

Here, we analyse whether the detectors are robust in classifying TGMs’ outputs from unseen text domains. Similar to Section 5.5.1, we train or fine-tune the detectors on the human-written and machine-generated texts from one domain and evaluate them on the texts from the other domains. Consider an example for the news domain, where we train the detectors on a mixture of human-written and machine-generated texts from only the news domain, and evaluate the detectors on six mixtures of human-written and machine-generated texts from the domains of RNC, social media, Wikipedia, digitalised diaries (Prozhito), strategic documents (Minek), and news.

Splits

CoAT is split into train, validation, and private test sets in the 70/10/20 ratio in a stratified fashion: 30.8k/4.4k/10.2k (strategic documents), 34.7k/4.9k/8.5k (news articles), 17.2k/2.4k/4.9k (digitalised diaries), 36.3k/5.2k/8.5k (Russian National Corpus), 12.5k/1.7k/2.9k (social media), and 28.8k/4k/9.9k (Wikipedia).

Figure 6. Results of testing the detectors’ robustness towards unseen text domains. Notations: Minek=strategic documents; Prozhito=digitalised diaries; RNC = Russian National Corpus.

Results

Figure 6 presents the results of this experiment. Overall, the detectors demonstrate similar transferred performance across all domains. The in-domain performance is up to 90% regardless of the detectors’ number of weight parameters and architecture. The detection is more reliable when training on texts from News and Wikipedia as indicated by less spiky patterns in the corresponding figures showing results for the models trained on these subsets. However, training on texts from Minek, Prozhito, and Social Media may result in near-random performance on the out-of-domain test sets, as corresponding figures show a single outstanding peak. We also observe that ruRoBERTa receives higher transferred accuracy than the other detectors, as seen from the purple dash-dotted line being on top of other lines in almost all patterns.

Discussion

While the TGMs are rapidly proliferating in different areas of life, the related ATD research primarily focuses on one particular domain (see Section 2.1). The single-domain evaluation limits the analysis of the detectors’ limitations. Several works report that existing detectors exhibit inconsistent multi-domain performance (Bakhtin et al. Reference Bakhtin, Gross, Ott, Deng, Ranzato and Szlam2019; Kushnareva et al. Reference Kushnareva, Cherniavskii, Mikhailov, Artemova, Barannikov, Bernstein, Piontkovskaya, Piontkovski and Burnaev2021). To our knowledge, our work is among the first to analyse the detectors’ cross-domain generalisation. We empirically show that the detectors fail to transfer when trained on specific domains, such as strategic financial documents and social media posts.

Crowd-sourcing annotation

Responses of human annotators are collected and stored anonymously. The average annotation pay is double the hourly minimum wage in Russia. The annotators are warned about potentially sensitive topics in data (e.g., politics, culture, and religion).

Social and ethical risks

The scope of risks associated with the misuse of generative language models is widely discussed in the community (Weidinger et al. Reference Weidinger, Mellor, Rauh, Griffin, Uesato, Huang, Cheng, Glaese, Balle and Kasirzadeh2021; Bommasani et al. Reference Bommasani, Hudson, Adeli, Altman, Arora, von Arx, Bernstein, Bohg, Bosselut and Brunskill2021). This problem has been addressed from the perspective of responsible artificial intelligence development: researchers and developers create new regulations and licenses (Contractor et al. Reference Contractor, McDuff, Haines, Lee, Hines, Hecht, Vincent and Li2022), require outputs from the TGMs to be marked as “generated,” and propose “watermarking” techniques to determine generated content (Kirchenbauer et al. Reference Kirchenbauer, Geiping, Wen, Katz, Miers and Goldstein2023). While our goal is to propose a novel large-scale ATD resource for the Russian language, we understand that the results of our work can be used maliciously, e.g., to reduce the performance of the detectors. However, we believe that CoAT will contribute to the development of more generalisable detectors for non-English languages.

Data collection

Learnable artificial text detection methods require human-written and machine-generated texts. The design of the ATD resources is inherently limited by the availability of diverse text domains and generative language models. In particular, such resources may suffer from decreasing inclusivity of the models due to the rapid development of the field of natural language generation. While we have addressed the diversity of the corpus in terms of the TGMs and text domains, it is crucial to continuously update CoAT to keep it up-to-date with the recent TGMs and conduct additional evaluation experiments.

Decoding strategies

The choice of the decoding strategy affects the quality of generated texts (Ippolito et al. Reference Ippolito, Kriz, Sedoc, Kustikova and Callison-Burch2019) and the performance of artificial text detectors (Holtzman et al. Reference Holtzman, Buys, Du, Forbes and Choi2020). The design of CoAT does not account for the diversity of decoding strategies, limiting the scope of the detectors’ evaluation. We leave this aspect for future work.