2.2.1 ALBERT: A lite version of BERT

Overall, increasing the size of a pretrained neural language model positively impacts subsequent tasks. However, growing the model becomes unfeasible at some point due to GPU/TPU memory limitations and huge training time. ALBERT (Lan et al. Reference Lan, Chen, Goodman, Gimpel, Sharma and Soricut2020) addressed these problems with two techniques for reducing parameters. This manuscript relies on an ALBERT model to extract knowledge from its rich contextualized representations. Although any other MLM model could have been selected, we adopt ALBERT because it uses fewer computational resources when compared to a BERT of the same size.

The architectural skeleton of ALBERT is similar to BERT, which means that it uses a transformer encoder with GELU activation function (Hendrycks and Gimpel Reference Hendrycks and Gimpel2016). ALBERT makes three main contributions to BERT design choices, as follows. Following BERT notation, $E$ is the size of the vocabulary vector representations, $L$ is the number of encoder layers, and $H$ is the hidden layers’ representation size.

4.2.1 Sequence-to-sequence learning architecture

As usual, we adopt the Seq2Seq encoder–decoder paradigm to solve the TST task. Formally, during learning, the model trains to generate an output sequence $Y = (y_1, \ldots, y_N )$ of length $N$ , conditioned on the input sequence $X = (x_1, \ldots, x_M)$ of length $M$ , where $x_i \in X$ and $y_i \in Y$ are tokens. The encoder–decoder neural network achieves the goal of generating the output sequence by learning a conditional probability distribution $P_{\theta }(Y|X)$ , by minimizing the cross entropy loss function $\mathcal{L}(\theta )$

(2) \begin{equation} \begin{split} \mathcal{L}(\theta ) = -\log P_{\theta }(Y|X) = - \sum _{t=1}^{N}\log P_{\theta }(y_t|y_{1:t-1},X) \end{split} \end{equation}

where $\theta$ are the model parameters.

Following (Dai et al. Reference Dai, Liang, Qiu and Huang2019), in this manuscript, the Transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) architecture is adopted as the Seq2Seq model. During the training and inference process, in addition to the embeddings of the input sentence $X$ , the model also receives as input an embedding of the target style. Hence, the input embeddings passed to the encoder are the target style embedding followed by the embeddings of the input sentence, which is the sum of the token embeddings and the positional embeddings, following the Transformer architecture. Thus, the method aims to learn a model that represents a probability distribution conditioned not only on $X$ but also on the desired target style $s^{\text{tgt}}$ . Thus, Equation (2) is modified to meet this characteristic, giving rise to the loss function

(3) \begin{equation} \begin{split} \mathcal{L}(\theta ) = -\log P_{\theta }(Y|X,s^{\text{tgt}}) = - \sum _{t=1}^{N}\log P_{\theta }\left(y_t|y_{1:t-1},X,s^{\text{tgt}}\right) \end{split} \end{equation}

4.2.2 Sequence-to-sequence pre-training

Pretraining and fine-tuning are usually adopted when target tasks have few examples available (Radford et al. Reference Radford, Narasimhan, Salimans and Sutskever2018; Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019). Several pretraining approaches adopt MLM, a kind of denoising auto-encoder trained to reconstruct the text where some random words have been masked. This kind of pretraining has mainly improved the performance of natural language understanding tasks, which can be justified by the fact that MLMs are composed only of a bidirectional encoder, while generation tasks adopt a left-to-right decoder. In this sense, the model BART (Lewis et al. Reference Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov and Zettlemoyer2020) combined bidirectional and auto-regressive transformers to show that sequence-to-sequence pretraining can benefit downstream language generation tasks.

Following this rationale, we pretrain our Seq2Seq model in a large collection of paraphrase pairs (Hu et al. Reference Hu, Singh, Holzenberger, Post and Van Durme2019). With that, we expect the model to learn the primary task of rewriting, allowing the generation of pseudo-parallel data and, consequently, handling the style transfer in a supervised fashion. We added another style embedding inside our style embedding layer to adapt this technique to our training framework. This way, besides the actual styles of the sentences, there is an additional one that we refer to as $s^{\text{para}}$ . It is the only style inserted into the model during the pretraining phase. During this phase, we minimize the following loss function

(4) \begin{equation} \begin{split} \mathcal{L}(\theta ) = -\log P_{\theta }(Y|X,s^{\text{para}}) = - \sum _{t=1}^{N}\log P_{\theta }\left(y_t|y_{1:t-1},X,s^{\text{para}}\right) \end{split} \end{equation}

4.3.1 Masked language model

One of the main contributions of this manuscript is to introduce the ability to transfer knowledge contained in the rich bidirectional contextualized representations provided by a MLM to a Seq2Seq model. MATTES adopts ALBERT (Lan et al. Reference Lan, Chen, Goodman, Gimpel, Sharma and Soricut2020) as the MLM, whose architecture is similar to the popular BERT model (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) but has fewer parameters.

From the language model learned by ALBERT, one can obtain the probability distribution of the masked tokens according to

(5) \begin{equation} \begin{split} P_{\phi }(X^m|X^u) = P_{\phi }(x_1^m, \ldots, x_l^m|X^u) \end{split} \end{equation}

where $x_*^m \in X^m$ are the masked tokens, $l$ is the number of masked tokens in $X$ , $X^u$ are the unmasked tokens, and $\phi$ are ALBERT parameters.

Before training the main model, we fine-tuned ALBERT with the available training dataset. Although ALBERT is prepared to handle a couple of sentences as inputs, given the problem definition, only one sentence is required. This is true either when fine-tuning or when training the main Seq2Seq model. Because of that, MATTES does not adopt ALBERT Sentence-order prediction loss component during the fine-tuning. When training the main TST model, ALBERT parameters do not change. As the next section explains, MATTES uses the probability distribution provided by ALBERT for each token of the input sentence as a label for training the model in one of the components of the loss function. Thus, instead of forcing the model to generate a probability distribution with the entire probability mass in a single token, the model is forced to have a smoother probability distribution, injecting probability mass into several tokens.

4.3.2 Knowledge distillation from the masked language model

This section details how we introduce the knowledge distillation strategy into the model loss function. To preserve the content of the original message, several previous TST methods adopt a back-translation (BT) strategy (Lample et al. Reference Lample, Subramanian, Smith, Denoyer, Ranzato and Boureau2019; Dai et al. Reference Dai, Liang, Qiu and Huang2019; He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020; Lai et al. Reference Lai, Toral and Nissim2021), proposed in Sennrich, Haddow, and Birch (Reference Sennrich, Haddow and Birch2016) for the machine translation task. BT generates pseudo-parallel sequences for training when no parallel sentences are available in the examples, thus generating latent parallel data. Thus, in the context of the TST task, pairs of sentences are created to train the model by automatically converting sentences from the training set to another style.

During training, the BT component takes a sentence $X$ and its style $s$ as input and converts it to a target style target $\hat{s}\neq s$ , generating the sentence $\hat{Y} = f_\theta (X, \hat{s})$ . After that, the generated sentence $\hat{Y}$ is passed as input to the model along with the original style $s$ , and the network is trained to learn to predict the original input sentence $X$ . The input sentence is first converted to the target style and then converted back to its original style.

The method proposed in Dai et al. (Reference Dai, Liang, Qiu and Huang2019) adopts the back-translation strategy to learn a TST model, minimizing the negative value of the logarithm of the probability that the generated sentence is equal to the original sentence with

(6) \begin{equation} \begin{split} \mathcal{L}_{\text{BT}}(\theta ) = -\log P_{\theta }(Y=X|f_\theta (X,\hat{s}),s) \end{split} \end{equation}

where $f_\theta (X, \hat{s})$ indicates the converted sentence and $s$ is the original input style.

Sequence-to-sequence models are normally trained from left to right. Thus, when generating each token to compose the sentence, the vocabulary probability distribution is conditioned only to the previous tokens in the sentence. This is to avoid each token seeing itself and the others after it. However, such an approach has the disadvantage of estimating the probability distribution only with the left context.

MATTES uses a configuration called masked transformer during training to overcome this limitation and generate a distribution that includes bidirectional information. The adoption of this architecture in the style transfer task comes from the observation that the probability distribution of a masked token $x_t^{m}$ given by an MLM contains both past and future context information. Thus, as no sentence pairs are available to perform supervised training, the central idea is to force MATTES to generate a distribution provided by ALBERT for each token. By doing that, MATTES smooths the probability curve and spreads probability mass over more tokens. Such additional information can improve the quality of the generated texts and, in particular, the style control of the model. Thus, during the optimization of the loss function component related to the back-translation (Equation 6), the target is no longer the distribution in which the entire probability mass is in a single token. Instead, it becomes the probability distribution given by the MLM $P_{\phi }(x_t^m|X^u)$ (Equation 5), for each token $x_t$ of the sentence input. The distribution provided by the MLM becomes a softer target for text generation during training, moving the model away from learning a more abrupt and unreal distribution, where the entire probability mass is in a single token.

Another point that MATTES leverages with MLM is the use of a knowledge distillation scheme (Hinton, Vinyals, and Dean Reference Hinton, Vinyals and Dean2015). Distilling consists of extracting the knowledge contained in one model through a specific training technique. This technique is commonly used to transfer information from a large, already trained model, a teacher, to a smaller model, a student, better suited for production. In such a scheme, the student uses the teacher’s output values as the goal, instead of relying on the training set labels. The distillation process controls better the optimization and regularization of the training process (Phuong and Lampert Reference Phuong and Lampert2019).

Several previous distillation methods train both the teacher and the student in the same task, aiming at model compression (Hinton et al. Reference Hinton, Vinyals and Dean2015; Sun et al. Reference Sun, Cheng, Gan and Liu2019). Here, we have a different goal, as we use distillation to take advantage of pretrained bidirectional representations generated through a MLM. Thus, ALBERT provides smoother labels to be used as targets during training in the knowledge distillation component of the Seq2Seq loss function, inspired by the method proposed in Chen et al. (Reference Chen, Gan, Cheng, Liu and Liu2020), to improve the quality of the generated text.

MATTES benefits from the distillation scheme by making the MLM assume the teacher’s role, while the unsupervised Seq2Seq model behaves like the student. Equation (7) shows how we adapt the back-translation component to be a bidirectional knowledge distillation component:

(7) \begin{equation} \begin{aligned} \mathcal{L}_{\text{bidi}}(\theta ) ={} & -\sum _{t=1}^{N} \sum _{w \in V}P_{\phi }(x_t = w |X^u) \cdot \\ &\cdot \log P_{\theta }(y_t = w |y_{1:t-1},f_\theta (X,\hat{s},s)) \end{aligned} \end{equation}

where $P_{\phi }(x_t)$ is the soft target provided by the MLM with learned parameters $\phi$ , $N$ is the sentence length, and $V$ denotes the vocabulary. Note that the parameters of the MLM are fixed during the training process. Figure 1 illustrates the learning process, where the objective is to make the probability distribution of the word $P_\theta (y_t)$ , provided by the student, be closer to the distribution provided by the teacher, $ P_\phi (x_t)$ .

Figure 1. Training illustration when the model is predicting the token $y_3$ using an MLM. $P_\theta$ is the student distribution, while $P_\phi$ is the teacher soft distribution provided by the MLM.

MATTES is trained with an adapted back-translation method that implements a knowledge distillation procedure:

(8) \begin{equation} \begin{split} \mathcal{L}_{\text{KD}}(\theta ) = \alpha \mathcal{L}_{\text{bidi}}(\theta ) + (1-\alpha )\mathcal{L}_{\text{BT}}(\theta ) \end{split} \end{equation}

where $\alpha$ is a hyperparameter to adjust the relative importance of the soft targets provided by the MLM and the original targets.

With the introduction of the new loss function term, the distribution is forced to become smoother during training. When the traditional one-hot representations are used as a target during training, the model is forced to generate the correct token. All other tokens do not matter to the model, treating equally tokens that would be more likely to occur and tokens with almost zero chance of occurring. In this way, by relying on a smoother distribution, the model increases its potential to generate more fluent sentences as the probability spreads over more tokens. Also, a smoother distribution avoids token generation entirely out of context and better controls the desired style.

4.4.1 Learning the discriminator network

The discriminator neural network is a multiclass classifier with $K+1$ classes. $K$ classes are associated with $K$ different styles, and the remaining class refers to the converted sentences generated by the masked Transformer. The discriminator network is trained to distinguish the original and the reconstructed sentences from the translated sentences. It means the classifier is trained to classify a sentence $X$ with style $s$ and its reconstructed sentence $Y$ , as belonging to the class $s$ , and the translated $\hat{Y}$ as belonging to the class of translated sentences. Accordingly, its loss function is defined as

(9) \begin{equation} \begin{split} \mathcal{L}_{\text{discriminator}}(\rho ) = -\log P_{\rho }(c|X) \end{split} \end{equation}

where $\rho$ are the parameters of the discriminator network, $c$ is the stylistic domain of the sample that can assume $K+1$ categories. The parameters $\theta$ of the generator network are not updated when training the discriminator.

4.4.2 Learning the generator network

A reconstruction component, a knowledge distillation component, and an adversarial component compose the final loss function of the masked transformer, as follows.

Input sentence reconstruction component

When the model receives as input a sentence $X$ along with its style $s$ , the model must be able to reconstruct the original sentence. The following reconstruction component is added to the loss function to make the model achieve this ability:

(10) \begin{equation} \mathcal{L}_{\text{self}}(\theta ) = -\log P_{\theta }(Y=X|X,s) \end{equation}

During training, $\mathcal{L}_{\text{self}}$ is optimized in the traditional way, that is, from left to right, masking the future context, according to Equation (3). Despite being possible, this component does not adopt the technique of knowledge distillation as we would like to isolate the distillation to a single component and verify its benefit.

Knowledge distillation component

To extract knowledge from a pretrained MLM to improve the transductive process of converting a sentence from one style to another, we introduce the following knowledge distillation component

(11) \begin{equation} \begin{split} \mathcal{L}_{\text{KD}}(\theta ) = \alpha \mathcal{L}_{\text{bidi}}(\theta ) + (1-\alpha )\mathcal{L}_{\text{BT}}(\theta ) \end{split} \end{equation}

where $\mathcal{L}_{\text{BT}}$ is as Equation (6) and $\mathcal{L}_{\text{bidi}}$ is defined as Equation (7). With this, we expect to smooth the probability distribution of the style converter model, producing more fluent sentences resembling the target domain texts.

During training, to generate the knowledge distillation component (Equation 11) of our cost function, the translated sentence style $\hat{s}$ inserted into the generator network to obtain $\hat{Y}$ depends on whether we are training from scratch or using the generator network already pretrained on paraphrase data. In the former case, as we do not have the style $s^{\text{para}}$ , we convert it to the other existing style. On the latter, $\hat{Y}$ is created according to the paraphrase style $s^{\text{para}}$ . In both approaches, $\hat{Y}$ is inserted back into the network along with the original style $s$ to generate our final probability distribution $P_\theta$ . This architectural modification unlocks our model to handle multiple styles at once. This way, regardless of the number of style domains, during training, we translate to $s^{\text{para}}$ and then back to $s^{\text{src}}$ . In inference time, we first translate to $s^{\text{para}}$ and then to $s^{\text{tgt}}$ . Translating to the paraphrase style can be thought of as normalizing the input sentence, striping out stylistic information regarding the source style.

Adversarial component

When adopting the training from scratch approach, if the model is only trained with the sentence reconstruction and knowledge distillation components, it could quickly converge to learn to copy the input sentence, that is, stick to learning the identity function. Thus, to avoid this problematic and unwanted behavior, an adversarial component is added to the cost function to encourage texts converted to style $\hat{s}$ , different from the input sentence style $s$ , to get closer to texts from the style $\hat{s}$ .

In the scenario with the presence of paraphrase style, the adversarial component tries to modify the generator such that the generated paraphrase sentence $\hat{Y}$ is pushed to become similar to other existing styles of the training set, as long as they are different from $s^{\text{src}}$ .

Generalizing for both approaches, the converted sentence $\hat{Y}$ is inserted into the discriminator neural network and, during training, the probability of the generated sentence being of the style $s^{\text{tgt}}$ , such that $s^{\text{tgt}}$ $\neq$ $s^{\text{src}}$ and $s^{\text{tgt}}$ $\neq$ $s^{\text{para}}$ , is maximized through optimization of the loss function defined in Equation 12. The $\rho$ parameters of the discriminator network are not updated when training the generator network.

(12) \begin{equation} \begin{split} \mathcal{L}_{\text{adversarial}}(\rho ) = -\log P_{\rho }(c=s^{\text{tgt}}|f_\theta (X,\hat{s})) \end{split} \end{equation}

These three loss functions are merged, and the overall objective for the generator network becomes:

(13) \begin{equation} \begin{split} a_1\mathcal{L}_{\text{self}}(\theta )+a_2\mathcal{L}_{\text{KD}}(\theta )+\mathcal{L}_{\text{adversarial}}(\rho ) \end{split} \end{equation}

where $a_1$ and $a_2$ are hyperparameters to adjust each component’s importance in the loss function of the generator network.

4.4.3 Adversarial training

Generative adversarial networks (GANs) (Goodfellow et al. Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio2014) are differentiable generator neural networks based on a game theory scenario where a generator network must compete against an opponent. The generator network produces samples $x=g(z;\theta ^{(g)})$ . The discriminator network’s adversary aims to distinguish between samples from the training set and the generator network. Thus, during training, the discriminator learns to classify samples as real or artificially generated by the generator network. Simultaneously, the generator network tries to trick the discriminator and produces samples that look like they are coming from the probability distribution of the dataset. At convergence, the generator samples will be indistinguishable from the actual training set samples, and the discriminator network can be dropped.

In the context of this manuscript, the adversarial strategy can be better understood according to the approach adopted. When training from scratch, it aims at converting texts to the desired style without incurring the failure to copy the input text. When training from a pretrained paraphrase model, the adversarial technique tries to push the generated paraphrases toward different styles $s^{\text{tgt}}$ , such that $s^{\text{tgt}}$ $\neq$ $s^{\text{src}}$ and $s^{\text{tgt}}$ $\neq$ $s^{\text{para}}$ . The general training procedure consists of, repeatedly, performing $n_d$ discriminator training steps, minimizing $\mathcal{L}_{\text{discriminator}}(\rho )$ , followed by $n_f$ generator network training steps, minimizing $a_1\mathcal{L}_{\text{self}}(\theta )+a_2\mathcal{L}_{\text{KD}}(\theta )+\mathcal{L}_{\text{adversarial}}(\rho )$ , until convergence. During the discriminator training steps, only the $\rho$ parameters of the discriminator network are updated. Analogously, during the generator network training steps, only the parameters $\theta$ of the masked transformer are updated. Figure 2 illustrates the adversarial training adopted by MATTES.

Figure 2. Adversarial training illustration. $G$ denotes the generator network, while $D$ denotes the discriminator network.

When back-translating during training (Figure 1), MATTES do not generate the transformed sentence $\hat{Y}$ through sampling or greedy decoding from the transductive probability distribution provided by the generator network. This decision is because propagating the gradients backward through discrete stochastic operations is not differentiable, typically requiring techniques such as REINFORCE (Williams Reference Williams1992) or Gumbel-Softmax distribution approximation (Jang, Gu, and Poole Reference Jang, Gu and Poole2017), which suffer from high variance (He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020). To overcome this difficulty, the softmax distribution generated by the MATTES decoder is inserted again into the MATTES encoder, along with the original sentence style, as in Dai et al. (Reference Dai, Liang, Qiu and Huang2019). This configuration makes it possible to directly propagate the gradients from the discriminator and generator networks to the reverse model that generated the target style translated sentences.

Overall, the MATTES architectural scheme includes a four-layer transformer architecture with four attention heads in each layer, for both the encoder and the decoder, and the discriminator network, as in Dai et al. (Reference Dai, Liang, Qiu and Huang2019). The vector representations of tokens, hidden states, and token position in the sentence have 256 dimensions. When training from scratch, we have a style embedding layer with two 256-dimensional vectors representing the stylistic domains. When starting from the pretrained Seq2seq, we have three vectors as we append the paraphrase style to the set of styles. The target style vector is inserted into the encoder along with the embeddings of each token of the sentence.

In the experiments, we do not include the start sentence token, only the end sequence token. During training, we replaced this token embedding with the style embedding as the first input embedding of the sequence. Thus, the vectors representing the styles are also trained and are part of the model.

This section described a machine learning method that addresses the unsupervised TST task using a Seq2Seq model pretrained in a large amount of paraphrase data and an MLM inserted into the training process with a knowledge distillation configuration. The Seq2Seq pretraining phase allows the model to create synthetically generated paraphrase pairs on-the-fly during training to help the transduction process. The distillation schema aims to extract knowledge from the MLM and enrich the text-generated quality. To verify if the proposed method improves the performance of TST, we conduct an experimental evaluation in the next section.

5.1.1 Datasets and tasks

MATTES was evaluated in three transfer style tasks in the English language: author imitation, formality transfer, and polarity swap. The author imitation task consists of converting the style of a sentence to the style of a particular author. The aim is to generate a paraphrase of the original sentence but with a different textual style. To verify the ability of MATTES to perform this task, we rely on a set of 21,000 (twenty-one thousand) sentences from William Shakespeare’s plays, transcribed into modern English. As they are translations from English into English, this dataset is also an intra-linguistic translation. The author imitation task can be considered an adaptation, where a text is modified within the language. In this case, it is a sum of two processes: a temporal and a linguistic adaptation. This dataset was curated in Xu et al. (Reference Xu, Ritter, Dolan, Grishman and Cherry2012) and previously used in unsupervised textual style transfer jobs (He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020; Krishna et al. Reference Krishna, Wieting and Iyyer2020). The dataset is divided into training, validation, and testing sets. We adopt $s_1$ to denote modern English and $\mathcal{D}_1$ for the dataset referring to this style. In contrast, $s_2$ denotes Shakesperian English, and $\mathcal{D}_2$ denotes the domain of the sentences in Shakesperian English. We evaluate MATTES both to transfer sentences from $s_1$ to $s_2$ and from $s_2$ to $s_1$ . Although sentence pairs exist, they are not used in pairs during training but only to evaluate results.

In the formality transfer task, our goal is to change the formality of the sentence while preserving its meaning. For this task, we adopted the GYAFC corpus (Rao and Tetreault Reference Rao and Tetreault2018), which contains formal and informal sentences from two domains: Entertainment & Music (E&M) and Family & Relationships (F&R). For every sentence from validation and test sets, there are four human-written references. In the experiments, we use the most commonly used F&R domain. The dataset contains 51,967 training examples in each formal and informal class. The test set has 2351 examples. We denote the formal style as $s_1$ and its domain as $\mathcal{D}_1$ , while the informal style is denoted by $s_2$ and its domain is $\mathcal{D}_2$ . MATTES is evaluated both to transfer sentences from $s_1$ to $s_2$ and from $s_2$ to $s_1$ .

The polarity swap task consists of converting the sentence to a different sentiment, preserving its general theme. In this task, we rely on the YELP dataset, collected in Shen et al. (Reference Shen, Lei, Barzilay and Jaakkola2017), which includes establishment evaluations performed within the YELP application. The dataset contains 250,000 negative sentences and 380,000 positive sentences. To quantitatively assess the generalization skills of the transfer models, we rely on a test set with 1000 parallel sentences annotated by humans, introduced in Li et al. (Reference Li, Jia, He and Liang2018). We denote the positive sentiment style as $s_1$ and its domain as $\mathcal{D}_1$ , while the negative sentiment style is denoted by $s_2$ and its domain is $\mathcal{D}_2$ . Again, MATTES is evaluated both to transfer sentences from $s_1$ to $s_2$ and from $s_2$ to $s_1$ .

Although these tasks have been handled indiscriminately as TST tasks, they have crucial differences that might indicate they should be treated differently during modeling and evaluation. More specifically, in polarity swap, the content is not strictly preserved (the message is actually the opposite). The important is to preserve the overall topic. In author imitation and formality transfer, instead, the “translation” happens more at the style level, and content must remain the same. In YELP, we can see that words related to the theme are intended to be retained, while polarity words are expected to be modified or replaced. On the contrary, in author imitation and formality transfer, the modifications happen at the stylistic level, affecting even noun phrases, but the essential message should be conserved. Since these two tasks can be seen much more as a rewriting task than the polarity swap, we expect the paraphrase pretraining strategy to benefit them more.

5.1.2 Baselines

We compare MATTES to recent models that made available the transferred test suite sentences. In the author imitation task, the results were compared with the Deep Latent Sequence (DLSM) (He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020) and STRAP (Krishna et al. Reference Krishna, Wieting and Iyyer2020) models, which showed relevant results in the metrics of content preservation. As the Style Transformer (Dai et al. Reference Dai, Liang, Qiu and Huang2019) model did not perform this task, we built an in-house implementation of this model so that its output samples could be compared with MATTES. Regarding the polarity swap task, we compare MATTES with the results of other models that obtained state-of-the-art (SOA) results for this task, including again the DLSM (He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020) and Style Transformer (Dai et al. Reference Dai, Liang, Qiu and Huang2019), in addition to the model proposed in Lai et al. (Reference Lai, Toral and Nissim2021) and also the RETRIEVEONLY, RULE-BASED, and DELETEANDRETRIVE models, which obtained the best results in the experiments published in Li et al. (Reference Li, Jia, He and Liang2018). For the formality transfer task, we also compared with the SOTA models that made their outputs available (Luo et al. Reference Luo, Li, Zhou, Yang, Chang, Sun and Sui2019; Yi et al. Reference Yi, Liu, Li and Sun2020; Lai et al. Reference Lai, Toral and Nissim2021).

5.1.3 Metrics for quantitative evaluation

Developing reliable automatic evaluation metrics for NLG tasks that reflect human judgment accordingly is still an open research field. However, automatic evaluation is cheaper and faster to run than human evaluation. Thus, first, we gather the most used automatic evaluation metrics from the literature (Yang et al. Reference Yang, Hu, Dyer, Xing and Berg-Kirkpatrick2018; Lample et al. Reference Lample, Subramanian, Smith, Denoyer, Ranzato and Boureau2019; He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020) focusing on two dimensions that effective textual transfer style systems must preserve.

Considering that content preservation is the most desired feature to style transfer, three distinct metrics were used to evaluate the models according to this dimension: BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), BartScore (Yuan et al. Reference Yuan, Neubig and Liu2021) and semantic similarity (SIM) (Wieting et al. Reference Wieting, Berg-Kirkpatrick, Gimpel and Neubig2019). The values of the three metrics are calculated using the sentence generated by the model and a previously annotated reference sentence. For the polarity swap task, as we only have annotated sentences for the test set, during the evaluation part of the training, we use the input as the reference for calculating the metrics.

BLEU (Bilingual Evaluation Understudy Score) is a fast and inexpensive metric widely used in NLP tasks, initially proposed for machine translation. BLEU is widely used in many languages and correlates well with human judgments. Although BLEU is traditionally used for inter-linguistic translation, it was adopted in this study to evaluate the adaptations of Shakespeare’s texts, which consist of an intra-linguistic translation task. In the experiments, BLEU was calculated using the NLTK (Bird Reference Bird2006) package. BLEU is calculated as

(14) \begin{equation} \begin{split} p_n &= \frac{\# n\textrm{-grams in the candidate found in the reference}}{\# n\textrm{-grams in the candidate}} \\ \text{BP} &= e^{\min (0,1-\frac{r}{c})} \\ \text{BLEU} &= \text{BP} \times \prod _{i=1}^{k}p_n^{w_n} \end{split} \end{equation}

where $c$ is the length of the translated candidate sentence, $r$ is the length of the reference sentence, $BP$ is a term that penalizes the difference between the length of the candidate and reference sentences, $k$ is the maximum n-gram one wants to evaluate, and $p_n$ is the precision score for the grams of length $n$ . According to the values proposed in Papineni et al. (Reference Papineni, Roukos, Ward and Zhu2002), we adopt $k=4$ and uniform weights $w_n = 1/N$ .

Wieting et al. (Reference Wieting, Berg-Kirkpatrick, Gimpel and Neubig2019) introduced a metric called SIM to measure semantic similarity between sentences. SIM aims to overcome the limitations of the BLEU metric and avoid giving partial credit and penalizing semantically correct candidates when they differ lexically from the reference sentence. For that, a coding model $g$ that tries to maximize the similarity of pairs of sentences present in a dataset of paraphrases (Wieting and Gimpel Reference Wieting and Gimpel2018) was trained. The $g$ encoder averages the vector representations of each sentence token to create a vector representation of the sentence. The similarity between a pair of sentences $ < X,\hat{X}>$ is obtained by encoding the two sentences with $g$ and then calculating the cosine similarity of the two representations:

(15) \begin{equation} \begin{split} \textrm{SIM} = \cos (g(X), g(\hat{X})) \end{split} \end{equation}

BARTScore (Yuan et al. Reference Yuan, Neubig and Liu2021) is a recently proposed metric tailored for generation tasks. It evaluates generated text from different perspectives, for example, informativeness and fluency. Although simple, BARTScore has been shown to correlate better with human judgments in different generation tasks and achieved the best performance on 16 of 22 settings against existing top-scoring metrics. Mathematically, BARTScore is the log probability of one text $\textbf{y}$ given another text $\textbf{x}$ .

(16) \begin{equation} \begin{split} \text{BARTScore} = \frac{1}{N} \sum _{t=1}^{N} \log p(y_t|y_{1:t-1},\textbf{x},\theta ) \end{split} \end{equation}

Regarding attribute control, following previous work (Shen et al. Reference Shen, Lei, Barzilay and Jaakkola2017; Yang et al. Reference Yang, Hu, Dyer, Xing and Berg-Kirkpatrick2018; Luo et al. Reference Luo, Li, Zhou, Yang, Chang, Sun and Sui2019; He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020; Lai et al. Reference Lai, Toral and Nissim2021), we train a neural convolutional classifier (Kim Reference Kim2014) to measure the extent to which the style is controlled. We train one classifier for each task to determine the stylistic domain to which a sentence belongs. The style control metric for each task is the accuracy this classifier gives to the generated sentences. The training set used to train the classifiers is the same one used during our main training. The classifiers have an accuracy of 82.7%, 85.1%, and 97.1% on the validation sets of Shakespeare, GYAFC and YELP, respectively. On the test sets, the values are 81.4%, 86.2%, and 97.0%, respectively.

Finally, as the overall score for model selection and for comparison to previous work (Luo et al. Reference Luo, Li, Zhou, Yang, Chang, Sun and Sui2019; Lai et al. Reference Lai, Toral and Nissim2021), we compute the harmonic mean (HM) of style accuracy and BLEU.

To select our best model, we settled thresholds for the style accuracy, and the model with the highest harmonic mean (HM) was selected to run on the test set. Since it is necessary to provide a greater weight to the content preservation metric, the HM suits well for our case. The threshold is necessary to avoid selecting models with a high degree of input copying, achieving a high harmonic mean even with a shallow style accuracy, and failing to control the desired style. For author imitation, we empirically established 66.6% accuracy, which is a bottom limit for style control, considering each transfer direction. For formality transfer and polarity swap, this limit was set to 70%.

5.1.4 Human evaluation

As the automatic metrics only give us shallow perceptions of the quality of the translated sentences, we compare the transfer quality across state-of-the-art models and ours by a small-scale human study. We conduct a human evaluation on the test set of both the YELP dataset and the GYAFC corpus. We left the Shapespeare domain out since it is more difficult for non-native speakers and non-experts to evaluate. For each dataset, we randomly select 48 samples for the human evaluation (24 for each style attribute). Each sample contains the transformed sentences generated by different models given the same source sentence. Annotators were asked to rate each output on three criteria on a Likert scale from 1 to 5: style control, content preservation, and an overall quality score. The annotators also rate human references for each dataset.

5.1.5 Hyperparameters and training details

During the experiments, some combinations of hyperparameters were evaluated according to the performance of MATTES in the validation set. The term $\alpha$ in the loss function of the knowledge distillation component (Equation 11) varied within the values $\{0.1, 0.5, 0.65\}$ . The knowledge distillation temperature $T_{\text{KD}}$ also varied by $\{1, 5, 10\}$ . Values for the number of steps of the discriminator $n_d$ and the number of steps of the generator $n_f$ were also experimented. The tuple ( $n_d$ , $n_f$ ) varied with the values $\{(7.5), (9.5), (10.5)\}$ . The contribution of each component to the MATTES loss function (Equations 10, 11, and 12) also varied ( $a_1\mathcal{L}_{\text{self}}(\theta )+a_2\mathcal{L}_{\text{KD}}(\theta )+\mathcal{L}_{\text{adversarial}}(\rho )$ ). As Dai et al. (Reference Dai, Liang, Qiu and Huang2019), it was concluded that executing random dropout on the input sentence tokens during the sentence reconstruction step positively impacts the model results. The dropout rate varied in $\{0.2, 0.3, 0.4\}$ .

We perform all the experiments using the Python programming language with the PyTorch (Paszke et al. Reference Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin, Desmaison, Antiga and Lerer2017) framework. The ALBERT implementation uses the HuggingFace transformers (Wolf et al. Reference Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu, Scao, Gugger, Drame, Lhoest and Rush2020) framework. Before the main MATTES training, for each task performed, we selected the available model albert-large-v2, and fine-tuned two ALBERT models, one for each stylistic domain, using only the traditional MLM task. In this adaptive domain pretraining step, as in Gururangan et al. (Reference Gururangan, Marasovic, Swayamdipta, Lo, Beltagy, Downey and Smith2020), we train each ALBERT model for 100 epochs, using the AdamW with $\epsilon$ equal to $1\mathrm{e}{-6}$ , linear learning rate with a warm-up, and maximum learning rate of $1\mathrm{e}{-5}$ . ALBERT does not use dropout or regularization during training. Training took place on four NVIDIA P100 GPUs with a training batch of eight examples per GPU. After fine-tuning ALBERT in each stylistic domain, the logitsFootnote ^b were extracted for each token of each sentence in the training set. Following Chen et al. (Reference Chen, Gan, Cheng, Liu and Liu2020), for saving computational resources, only the top-8 logits were considered to be used as labels during the main MATTES training.

To take advantage of available generic data, we pre-trained our Seq2seq model on pairs of paraphrase (Hu et al. Reference Hu, Singh, Holzenberger, Post and Van Durme2019) and used the trained model as starting point for the main adversarial training. The training details followed Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017). We trained the model for 300,000 steps, using the Adam optimizer (Kingma and Ba Reference Kingma and Ba2015) with $\beta _1 = 0.9$ , $\beta _2 = 0.98$ and $\epsilon$ = $1\mathrm{e}{-9}$ . Each training step took about $0.12$ seconds, and the whole pre-training was about $20$ hours on a single P100 GPU. We apply dropout with a rate of $P_{\text{dropout}}=0.1$ . As in Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), we varied the learning rate throughout training, increasing the learning rate linearly for the first 4000 steps and decreasing it proportionally to the inverse square root of the step number. From the complete PARABANK 2 (Hu et al. Reference Hu, Singh, Holzenberger, Post and Van Durme2019) containing 19.7M pairs of paraphrases, we filtered out pairs in which at least one of the sentences had less than six words or more than 50 words. We split the remaining pairs into training and validation sets. Our final paraphrase training set reached 11.4M pairs, and the validation set 2.8M.

All primary training experiments were run on Tesla P100-SXM2 GPUs using the Ubuntu operating system on a machine with an Intel(R) Xeon(R) CPU E5-2698 v4. On average, the Yelp dataset takes 30 hours with a training batch of 192, the GYAFC corpus takes 15 hours with a training batch of 128, while the Shakespeare dataset takes 15 hours with a training batch of 64. This time drops by approximately half when starting from the pretrained Seq2Seq model.

5.2.1 Quantitative results

Table 2 shows the results using the automatic metrics. In the YELP dataset, the best models displayed in Li et al. (Reference Li, Jia, He and Liang2018) were compared and complemented with Yi et al. (Reference Yi, Liu, Li and Sun2020), He et al. (Reference He, Wang, Neubig and Berg-Kirkpatrick2020), Dai et al. (Reference Dai, Liang, Qiu and Huang2019), and Lai et al. (Reference Lai, Toral and Nissim2021). In the Shakespeare dataset, MATTES results were compared to the results of He et al. (Reference He, Wang, Neubig and Berg-Kirkpatrick2020) and Krishna et al. (Reference Krishna, Wieting and Iyyer2020). In addition, the results were also compared with our implementation of the style transformer (Dai et al. Reference Dai, Liang, Qiu and Huang2019) model in the author imitation task, as the original work did not include this task. This implementation is similar to MATTES but uses the traditional back-translation component in the loss function rather than the knowledge distillation component. In the GYAFC corpus, MATTES results were compared to the results of Luo et al. (Reference Luo, Li, Zhou, Yang, Chang, Sun and Sui2019), Yi et al. (Reference Yi, Liu, Li and Sun2020), and Lai et al. (Reference Lai, Toral and Nissim2021), the state-of-the-art model. We also compared it with our implementation of the style transformer (Dai et al. Reference Dai, Liang, Qiu and Huang2019) model.

Table 2. Results with automatic evaluation metrics. BScore stands for BARTScore, and HM is the harmonic mean of BLEU and accuracy. The best results are in bold.

In Shakespeare’s dataset, MATTES surpassed all previous approaches regarding all content preservation metrics. It also achieved the best overall performance (HM) on this task. The Deep Latent Sequence Model (DLSM) (He et al. Reference He, Wang, Neubig and Berg-Kirkpatrick2020) has the best style control metric, however, at the expense of low values in the content preservation metrics. On the other hand, we can increase the style accuracy threshold if we desire more control over the style. In Appendix D, we show models selected using a higher threshold.

In the YELP dataset, MATTES achieved the second-highest results for all the content preservation metrics and the overall score, staying very close to the best model. Nevertheless, the style control metric was the worst among all the compared models.

In the GYAFC corpus, MATTES achieves the highest value for style control. As in YELP, it achieves the second-highest results for all the content preservation metrics and the overall score. As we can see in Section 5.2.3, the style control result is mainly due to the knowledge distillation technique.

By examining the results, we can see that our full model is quite effective in preserving the content of the input while keeping a reasonable style accuracy in all tasks. In the author imitation task and in the formality transfer task, our approach achieved new state-of-the-art results in terms of content preservation and style control, respectively. In Section 5.2.3, we evaluate the impact of our proposed training strategy; specifically, we want to quantify the impact of the Seq2Seq pretraining phase and the knowledge distillation training technique.

Statistical significance analysis

We conducted statistical tests to compare the BARTScore and SIM metrics achieved by MATTES and the other two models that obtained the best content preservation metrics. We left out BLEU since it is a more limited metric that measures lexical matching only. We denote $H_0$ as the null hypothesis and $H_1$ as the alternative hypothesis to be tested. The significance level $\alpha = 0.05$ was adopted, and we assume that the random variables $\bar{B}$ and $\bar{S}$ , which correspond, respectively, to the population mean of BARTScore and SIM, have a normal distribution for each model. The tests were based on the sample means of the two metrics in the test set. Under the null hypothesis, the p-value is the probability of obtaining a statistic equal to or more extreme than that observed in a sample. Thus, when a p-value smaller than the significance level is obtained, this indicates an unlikely event, indicating rejection of $H_0$ .

The results of the statistical tests in Table 3 reinforced the results of Table 2. In the author imitation task, MATTES outperformed the other baseline models on both metrics. In the polarity swap, MATTES outperforms the Style Transformer Model on both metrics. When comparing with the SOTA model (Lai et al. Reference Lai, Toral and Nissim2021), for a confidence interval of $95\%$ , it is not possible to affirm the superiority of any model regarding both metrics. In the formality transfer task, Lai et al. (Reference Lai, Toral and Nissim2021) is statistically superior in both metrics, and MATTES confirms the superiority over the style instance model only for the BARTScore metric.

Table 3. T-test over the mean population for BARTSore and SIM metric on the test set

5.2.2 Human evaluation results

Due to the limitations of automatic metrics, we also conduct human evaluations on YELP and GYAFC datasets. Table 4 shows the evaluation of the style transfer accuracy, content preservation, and an overall quality score obtained by MATTES and the compared models. All three aspects are rated with a range of 1–5. The results are in line with our automatic evaluations and add confidence to the efficacy of our proposed techniques in achieving style transfer across multiple dimensions. Our scores are around the state-of-the-art values regarding both style control and content preservation in the formality transfer and polarity swap tasks, respectively. Additionally, it is worth mentioning the high scores achieved by the baseline Lai et al. (Reference Lai, Toral and Nissim2021), mainly for the content preservation metric, in some cases even surpassing the scores reached by the references which are sentences created by humans.

Table 4. Results from the human evaluation on Yelp and GYAFC datasets. The best results are in bold.

5.2.3 Ablation studies

In this section, we measure the individual impact of the pretraining phase and the model training components. We also show how the content preservation metrics correlate between them.

Paraphrase pre-training and model components

To confirm that TST models can benefit from our proposals, we conducted an ablation study on all datasets by eliminating or modifying a certain component (e.g., objective functions or pre-training phase). We tested the following variations: (1) FULL MODEL: our proposed model with all the components described in Section 4; (2) NO KD: the model without the knowledge distillation component ( $\alpha =0$ ); (3) NO PARA: the model trained from scratch, without pre-training the Seq2Seq model on paraphrase data; (4) NO PARA KD: the model trained from scratch and without knowledge distillation; (5) NO ADV: the model trained without the adversarial loss component; and (6) NO ADV KD: the model trained without the adversarial loss component and without the knowledge distillation ( $\alpha =0$ ). Table 5 exhibits the results.

Table 5. Evaluation results for the ablation study. Acc. stands for accuracy, BScore for BARTScore, and HM for the harmonic mean between BLEU and accuracy

The results indicate that the pretraining strategy significantly impacts content preservation. On all tasks, pretraining considerably increased the content preservation metric. In the Shakespeare dataset and in the GYAFC corpus, the style accuracy also improved with pre-training, which makes sense since the diversity of the paraphrase model should lead to a lower degree of input copying. On the other hand, in the YELP dataset, the pretraining approach harmed the model style control. This also makes sense since the polarity swap task is not a rewriting task, so having a paraphrase of a sentence does not help. For that, something that changes the meaning is necessary. This observation indicates that these tasks should not be indiscriminately tackled as TST tasks since there is a clear need to change the meaning of the polarity swap task.

Regarding the isolated impact of our proposed knowledge distillation technique, all the model variations from Table 5 trained with the distillation achieved higher accuracy and HM when compared with the variations trained without. Although adversarial training is the most crucial component for style control, the distillation also positively affects style accuracy. Comparing the two model variations without adversarial training (NO˙ADV and NO˙ADV˙KD), we realize distillation increases the accuracy, yet not so impressively as the adversarial component.

Since we decided to use the harmonic mean (HM) of the BLUE with the style accuracy for model selection, to interpret the impact of the pre-training phase better, in Figure 3 located in Appendix C, we plot these three metrics achieved by our FULL MODEL and its variation NO PARA along the training process on the Yelp validation set. From the graphics, we can realize that the training dynamics are different. When training the model from scratch, the content preservation metric starts almost from zero until reaching its limit. In contrast, the pretrained model starts at its maximum but with low accuracy. Once its highest value is reached, the HM tends downward after many training steps. At last, we notice the FULL MODEL reaches its highest HM faster than the model trained from scratch, showing another benefit from pretraining.

The hyperparameters of the models in Table 5 are listed in Appendix A.

Metric analysis

Since evaluating generated texts is an open field and a challenging task, we adopted three metrics for content preservation. To glean further insights about these choices, we calculate the Pearson Correlation between evaluation metrics for content preservation over N systems (Table 6).

Table 6. Pearson correlation between content preservation metrics over N systems

The results show that while SIM and BARTScore highly correlated with each other, BLEU does so to a lesser extent, suggesting this might be a less robust measure to assess the goodness of TST tasks. Since transfer style often involves lexical changes, n-gram-based matching metrics such as BLEU usually fail to recognize information beyond the lexical level. This result strengthens the necessity for adopting metrics concerned with capturing the overall semantics of a sentence, like SIM and BARTScore.

5.2.4 Qualitative analyses

At last, we carry out some qualitative analyses: (1) compare samples generated by MATTES and DLSM and (2) compare our model and training approach with the state-of-art method in the Yelp dataset (Lai et al. Reference Lai, Toral and Nissim2021).

MATTES versus deep latent sequence model

We examine the output from our model and the Deep Latent Sequence model for the author imitation task. We picked this task because the reference outputs for the test set are provided. From the qualitative analysis, we observed that both models can generate good translations, but MATTES tends to preserve the content better. On the other hand, Deep Latent Sequence tends to create too short sentences when the input sentence is long, losing content. This observation concurs with the fact that MATTES’s content preservation metric is the highest. The attention mechanisms might have helped in handling long sentences. Besides, the sentences generated by the Deep Latent Sequence model have lower perplexity than the test set, indicating short or trivial sentences. Table 7 exhibits examples of the generated sentences for long inputs.

Table 7. Transferred sentences for the author imitation task

Appendix B shows other examples of transferred sentences by MATTES, DLSM, and Style Transformer in the sentiment transfer task.