A) Spectral conversion with DNN

In the DNN, given an input spectral feature vector $\boldsymbol {X}_t$ at frame $t$, the conditional PDF of the target spectral feature vector $\boldsymbol {Y}_t$ is defined as

(1)\begin{equation} P(\boldsymbol{Y}_t|\boldsymbol{X}_t,\boldsymbol{\lambda}_S) = \mathcal{N}(\boldsymbol{Y}_t;f_{\boldsymbol{\lambda}_S}(\boldsymbol{X_t}),\boldsymbol{D}), \end{equation}

where $\mathcal {N}(\cdot; \boldsymbol {\mu },\mathbf {\Sigma })$ denotes a Gaussian distribution with a mean vector $\boldsymbol {\mu }$ and a covariance matrix $\mathbf {\Sigma }$. $\boldsymbol {D}$ is the diagonal covariance matrix of the target spectral features computed using the training dataset beforehand. The set of parameters of the network model is denoted as $\boldsymbol {\lambda }_S$, and $f_{\boldsymbol {\lambda }_S}(\cdot )$ is the feedforward function of the network.

In the training phase of the DNN network, to estimate the updated model parameters $\hat {\boldsymbol {\lambda }}_S$, backpropagation is performed throughout the network to minimize the loss function as follows:

(2)\begin{equation} \hat{\boldsymbol{\lambda}}_S=\arg\,\min_{\boldsymbol{\lambda}_S}\frac{1}{T}\sum_{t=1}^{T} \big(\boldsymbol{Y}_t-f_{\boldsymbol{\lambda}_S}(\boldsymbol{X}_t)\big)^{\top} \boldsymbol{D}^{-1}\big(\boldsymbol{Y}_t-f_{\boldsymbol{\lambda}_S}(\boldsymbol{X}_t)\big),\end{equation}

where $\top$ denotes the transpose operator and $T$ denotes the total number of time frames.

Then, using the trained DNN network model, a spectral conversion function can be applied to generate the estimated target spectral trajectory $\hat {\boldsymbol {y}}=[\hat {\boldsymbol {y}}_1^{\top },\hat {\boldsymbol {y}}_2^{\top },\dotsc ,\hat {\boldsymbol {y}}_t^{\top },\dotsc , \hat {\boldsymbol {y}}_T^{\top }]^{\top }$ by using the MLPG [Reference Toda, Black and Tokuda22, Reference Tokuda, Kobayashi and Imai53] procedure as follows:

(3)\begin{equation} \hat{\boldsymbol{y}} = (\boldsymbol{W}^{\top}\overline{\boldsymbol{D}}^{-1}\boldsymbol{W})^{-1} \boldsymbol{W}^{\top}\overline{\boldsymbol{D}}^{-1}\boldsymbol{M}, \end{equation}

where $\boldsymbol {M}=[f_{\boldsymbol {\lambda }_S}(\boldsymbol {X}_1)^{\top },f_{\boldsymbol {\lambda }_S}(\boldsymbol {X}_2)^{\top }, \dotsc ,f_{\boldsymbol {\lambda }_S}(\boldsymbol {X}_t)^{\top },\dotsc , f_{\boldsymbol {\lambda }_S}(\boldsymbol {X}_T)^{\top }]^{\top }$ is the sequence of network outputs, and $\overline {\boldsymbol {D}}^{-1}$ denotes a sequence of inverted matrices of the diagonal covariance $\boldsymbol {D}$. The transformation matrix is denoted as $\boldsymbol {W}$, which is used to enhance a sequence of spectral feature vectors with its delta feature contexts, i.e. the rate of change of values between successive feature frames.

B) Spectral conversion with DMDN

For DMDN-based [Reference Zen and Senior52] spectral conversion, given an input spectral feature vector $\boldsymbol {X_t}$ at frame $t$, the conditional PDF of the target spectral feature vector $\boldsymbol {Y}_t$ is defined as a mixture of distributions as

(4)\begin{equation} P(\boldsymbol{Y}_t|\boldsymbol{X}_t,\boldsymbol{\lambda})= \sum_{m=1}^{M}\alpha_{m,t,\boldsymbol{\lambda}}\mathcal{N}(\boldsymbol{Y}_t; \boldsymbol{\mu}_{m,t,\boldsymbol{\lambda}},\mathbf{\Sigma}_{m,t,\boldsymbol{\lambda}}), \end{equation}

where the Gaussian distribution is denoted as $\mathcal {N}(\cdot; \boldsymbol {\mu },\mathbf {\Sigma })$ with a mean vector $\boldsymbol {\mu }$ and a covariance matrix $\mathbf {\Sigma }$. The index of the mixture component is denoted by $m$ and the total number of mixture components is $M$. The set of model parameters is denoted as $\boldsymbol {\lambda }$. The time-varying weight, mean vector, and diagonal covariance matrix of the $m$th mixture component are, respectively, denoted as $\alpha _{m,t,\boldsymbol {\lambda }}$, $\boldsymbol {\mu }_{m,t,\boldsymbol {\lambda }}$, and $\mathbf {\Sigma }_{m,t,\boldsymbol {\lambda }}$. These time-varying mixture parameters are taken from the network output

\begin{align*} f_{\boldsymbol{\lambda}}(\boldsymbol{X}_t)& = [\alpha_{1,t,\boldsymbol{\lambda}},\dotsc, \alpha_{M,t,\boldsymbol{\lambda}},\boldsymbol{\mu}_{1,t,\boldsymbol{\lambda}}^{\top},\dotsc, \boldsymbol{\mu}_{M,t,\boldsymbol{\lambda}}^{\top},\\ & \qquad {\rm diag}(\boldsymbol{\Sigma}_{1,t,\boldsymbol{\lambda}})^{\top},\dotsc, {\rm diag}(\mathbf{\Sigma}_{M,t,\boldsymbol{\lambda}})^{\top}]^{\top}.\end{align*}

To estimate the updated model parameters $\hat {\boldsymbol {\lambda }}$ in the training of a DMDN-based spectral conversion model, backpropagation is performed according to the following negative log-likelihood function:

(5)\begin{align} \hat{\boldsymbol{\lambda}}& = \arg\,\min_{\boldsymbol{\lambda}}\frac{1}{T}\sum_{t=1}^{T}\sum_{m=1}^{M}D\log{2\pi}\notag\\ & \quad + \frac{1}{2}\log|\Sigma_{m,t,\boldsymbol{\mathbf{\lambda}}}|-\log{\alpha_{m,t,\boldsymbol{\lambda}}} \nonumber\\ & \quad -\frac{1}{2}(\boldsymbol{Y}_t-\boldsymbol{\mu}_{m,t,\boldsymbol{\lambda}})^{\top} \Sigma_{m,t\boldsymbol{\mathbf{\lambda}}}^{-1}(\boldsymbol{Y}_t-\boldsymbol{\mu}_{m,t,\boldsymbol{\lambda}}). \end{align}

Similar to the DNN, in the conversion phase, the estimated target spectral trajectory $\hat {\boldsymbol {y}}$ is generated by using the MLPG [Reference Toda, Black and Tokuda22, Reference Tokuda, Kobayashi and Imai53] procedure as follows:

(6)\begin{equation} \hat{\boldsymbol{y}} = \left(\boldsymbol{W}^{\top}\overline{{\boldsymbol{\Sigma}_{\hat{\boldsymbol{m}}}^{-1}}}\boldsymbol{W}\right)^{-1}\boldsymbol{W}^{\top} \overline{\Sigma_{\hat{\boldsymbol{m}}}^{-1}}{\boldsymbol{\mu}_{\hat{\boldsymbol{m}}}}, \end{equation}

where the transformation matrix used to append the spectral feature vector sequence with its delta features is denoted as $\boldsymbol {W}$. The sequence of mixture-dependent time-varying inverted diagonal covariance matrices is denoted as $\overline {\mathbf {\Sigma }_{\hat {\boldsymbol {m}}}^{-1}}$ and that of the mean vectors is denoted as $\overline {\boldsymbol {\mu }_{\hat {\boldsymbol {m}}}}$. The suboptimum mixture component sequence, which is denoted as $\hat {\boldsymbol {m}}=\{\hat {m}_1,\hat {m}_2,\dotsc ,\hat {m}_t,\dotsc ,\hat {m}_T\}$, used to develop the respective sequences of time-varying mixture parameters is determined as

(7)\begin{equation} \hat{\boldsymbol{m}} = {\rm argmax}_{\boldsymbol{m}}\prod_{t=1}^{T}\alpha_{m,t,\boldsymbol{\lambda}} \mathcal{N}\left(\boldsymbol{Y}_t;\boldsymbol{\mu}_{m,t,\boldsymbol{\lambda}},\Sigma_{m,t,\mathbf{\boldsymbol{\lambda}}}\right). \end{equation}

Note that both (3) and (6) generate the estimated $D$-dimensional target feature vector sequence $\hat {\boldsymbol {y}}$, whereas the input to the network is 2D input feature vector $\boldsymbol {X}_t$ at each time $t$. This is because the network actually outputs 2D feature vector for DNN, i.e. $f_{\boldsymbol {\lambda }_S}(\boldsymbol {X}_t)$, or $M$ sets of 2D mean vector $\boldsymbol {\mu }_{m,t,\boldsymbol {\lambda }}$ and diagonal covariance matrix $\rm {diag}(\mathbf {\Sigma }_{m,t,\boldsymbol {\lambda }})$ for DMDN at each time $t$, where they were then processed by the MLPG procedure to generate the spectral trajectory by considering their temporal correlation. This kind of explicit treatment can be alleviated by directly incorporating the use of network architecture with temporal modeling capabilities, such as RNN, which is described in the following section.

C) Spectral conversion with RNN

It is also important to note that a speech signal, and thus its parametrization, is time-sequence data. Therefore, it would be more reasonable to develop a spectral conversion model that is capable of capturing the dependencies within a sequence of spectral feature vectors, such as by using the RNN architectures. Hence, we propose the use of LSTM [Reference Hochreiter and Schmidhuber36] and a GRU [Reference Cho37] as the basis RNN units used to capture the long-term context dependencies on the input spectral features. The flow of the RNN-based spectral modeling is shown in Fig. 2. It can be observed that additional convolutional input layers are also used to extract better contextual input frames. Note that, different from the previous DNN/DMDN architectures, in the proposed RNN-based spectral conversion models, all of the components, except for the normalization and denormalization layers (which are kept fixed to make the network work with the normalized values of the features), are optimized during training.

Fig. 2. Spectral mapping model with RNN-based architectures.

1) LSTM-based spectral conversion model

Given an input spectral feature vector $\boldsymbol {x}_t$ at frame $t$, the estimated target spectral feature vector $\hat {\boldsymbol {y}}_t$ is estimated as

(8)\begin{equation} \hat{\boldsymbol{y}}_t = \boldsymbol{W}_{hy}\boldsymbol{h}_t + \boldsymbol{b}_{y}, \end{equation}

where $\boldsymbol {W}_{hy}$ and $\boldsymbol {b}_{y}$, respectively, denote the weights and biases of the output layer in Fig 2. The hidden state $\boldsymbol {h}_t$ is produced by the LSTM units [Reference Hochreiter and Schmidhuber36] as follows:

(9)\begin{align} \boldsymbol{i}_t & = \sigma(\boldsymbol{W}_{xi}[\boldsymbol{x}_t^{\top},\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} +\boldsymbol{W}_{hi}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{i}) \end{align}

(10)\begin{align} \boldsymbol{f}_t & = \sigma(\boldsymbol{W}_{xf}[\boldsymbol{x}_t^{\top},\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} +\boldsymbol{W}_{hf}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{f}) \end{align}

(11)\begin{align} \boldsymbol{g}_t & = \tanh(\boldsymbol{W}_{xg}[\boldsymbol{x}_t^{\top},\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} +\boldsymbol{W}_{hg}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{g}) \end{align}

(12)\begin{align} \boldsymbol{o}_t & = \sigma(\boldsymbol{W}_{xo}[\boldsymbol{x}_t^{\top},\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} +\boldsymbol{W}_{ho}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{o}) \end{align}

(13)\begin{align} \boldsymbol{c}_t & = \boldsymbol{f}_{t}\odot\boldsymbol{c}_{t-1} + \boldsymbol{i}_t\odot\boldsymbol{g}_t \end{align}

(14)\begin{align} \boldsymbol{h}_t & = \boldsymbol{o}_{t}\odot\tanh(\boldsymbol{c}_t) \end{align}

where $\hat {\boldsymbol {y}}_0 = \boldsymbol {0}$. The cell state is denoted as $\boldsymbol {c}_t$. The input, forget, cell, and output gates are denoted as $\boldsymbol {i}_t$, $\boldsymbol {f}_t$, $\boldsymbol {g}_t$, and $\boldsymbol {o}_t$, respectively. The trainable weights and biases are denoted by the corresponding $\boldsymbol {W}$ and $\boldsymbol {b}$, respectively. $\sigma$ denotes a sigmoid function and $\odot$ denotes an element-wise product.

The network model parameters $\hat {\boldsymbol {\lambda }}_\textrm {R}$ are optimized through backpropagation for the loss function, computed by considering the distortions in the mel-cepstrum domain [Reference Mashimo, Toda, Shikano and Campbell54], as follows:

(15)\begin{equation} \hat{\boldsymbol{\lambda}}_\textrm{R} = \arg\,\min_{\boldsymbol{\lambda}_\textrm{R}}\frac{1}{T}\sum_{t=1}^{T} \frac{10\sqrt{2}}{\ln{10}}\sum_{d=1}^{D}|\hat{y}_t(d)-y_t(d)|, \end{equation}

where $|\cdot |$ denotes the absolute function. The $d$th dimension estimated spectral feature at time $t$ is denoted as $\hat {y}_t(d)$, where the estimated spectral feature vector $\hat {\boldsymbol {y}}_t$ can be written in terms of the feedforward function in an RNN-based spectral mapping model as $f_{\boldsymbol {\lambda }_\textrm {R}}(\boldsymbol {x}_t)=[\hat {y}_t(1),\hat {y}_t(2),\dotsc ,\hat {y}_t(d),\dotsc ,\hat {y}_t(D)]^{\top }$. Note that the input source spectra $\boldsymbol {x}_t$ is firstly fed into the input convolutional layers before being fed into the RNN units as shown in Fig. 2.

2) GRU-based spectral conversion model

In addition to modeling the long-term context dependencies, it is also worthwhile considering a more compact model architecture, such as the GRU [Reference Cho37] units, which will be more suitable for our purpose in using a small amount of training data. A more compact network will also be more suitable for real-time applications, which require low-latency computation.

In the proposed GRU-based spectral conversion model, given an input spectral feature vector $\boldsymbol {x}_t$ at frame $t$, the following functions are computed by the GRU units:

(16)\begin{align} \boldsymbol{r}_t & =\sigma(\boldsymbol{W}_{xr}[\boldsymbol{x}_t,\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} + \boldsymbol{W}_{hr}\boldsymbol{h}_{t-1} + \boldsymbol{b}_{r}) \end{align}

(17)\begin{align} \boldsymbol{z}_t & =\sigma(\boldsymbol{W}_{xz}[\boldsymbol{x}_t,\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} + \boldsymbol{W}_{hz}\boldsymbol{h}_{t-1} + \boldsymbol{b}_{z}) \end{align}

(18)\begin{align} \boldsymbol{n}_t & =\tanh(\boldsymbol{W}_{xn}[\boldsymbol{x}_t,\hat{\boldsymbol{y}}_{t-1}^{\top}]^{\top} +\boldsymbol{r}_t\odot(\boldsymbol{W}_{hn}\boldsymbol{h}_{t-1})+\boldsymbol{b}_{n}) \end{align}

(19)\begin{align} \boldsymbol{h}_t & =(1-\boldsymbol{z}_t)\odot\boldsymbol{n}_t + \boldsymbol{z}_t\odot\boldsymbol{h}_{t-1}, \end{align}

where $\hat {\boldsymbol {y}}_t$ is given by (8) and, similarly, $\hat {\boldsymbol {y}}_0=\boldsymbol {0}$. The hidden state of the GRU is denoted as $\boldsymbol {h}_t$. The reset, update, and new gates are denoted as $\boldsymbol {r}_t$, $\boldsymbol {z}_t$, and $\boldsymbol {n}_t$, respectively. The corresponding weights and biases are, respectively, denoted by $\boldsymbol {W}$ and $\boldsymbol {b}$. The sigmoid function is denoted by $\sigma$ and $\odot$ denotes an element-wise product. As in the LSTM-based model, for multiple layers of GRU units, the hidden state of the current layer is used as the input for the succeeding layer. As can be observed, the modeling of long-term context dependencies in GRU is achieved through fewer computation steps, and model parameters, compared to the LSTM, which might be better for limited data situations and real-time computation.

A) WN vocoder

WN [Reference van den Oord23] is a deep AR-CNN that is capable of modeling speech waveform samples for their previous samples. To efficiently increase the number of receptive fields of waveform samples, WN uses a stack of dilated CNNs using residual blocks, making it possible to produce human-like speech sounds. Moreover, when it is conditioned with naturally extracted speech features, such as spectral and excitation parameters, a WN vocoder [Reference Tamamori, Hayashi, Kobayashi, Takeda and Toda44–Reference Adiga, Tsiaras and Stylianou46] is capable of producing meaningful speech waveforms with the natural quality compared with the conventional vocoder framework.

Given a sequence of auxiliary feature vectors $\boldsymbol {h}=[\boldsymbol {h}_1^{\top },\boldsymbol {h}_2^{\top },\dotsc ,\boldsymbol {h}_t^{\top },\dotsc ,\boldsymbol {h}_T^{\top }]^{\top }$, the likelihood function of a sequence of the corresponding waveform samples $\boldsymbol {s}=[s_1,s_2,\dotsc ,s_t,\dotsc ,s_T]^{\top }$ is given by

(20)\begin{equation} P(\boldsymbol{s}|\boldsymbol{h},\boldsymbol{\lambda}) = \prod_{t=1}^{T}P(s_t|\boldsymbol{h}_t,\boldsymbol{s}_{t-p}), \end{equation}

where the conditional PDF of a waveform sample $P(s_t|\boldsymbol {h}_t,\boldsymbol {s}_{t-p})$ is modeled by the WN vocoder, and $\boldsymbol {s}_{t-p}$ denotes the previous samples with $p$ receptive fields. In the training, the ground-truth previous samples are given, i.e. teacher-forcing mode, whereas, in the synthesis phase, the waveform is generated sample-by-sample. Note that the modeling of the waveform samples in a WN vocoder can be performed as a classification problem, where the floating values of the 16-bit waveform will be discretized into 256 categories using the $\mu$-law algorithm. The inverse $\mu$-law algorithm is used in synthesis after sampling from the output distribution.

B) Postconversion processing in VC

In VC, to use the WN vocoder, estimated speech features, generated from a statistical mapping model, such as the ones described in Section III, are fed as auxiliary conditioning features in equation (20) to generate the converted speech waveform. However, in this case, the WN vocoder will face a significant degradation [Reference Kobayashi, Hayashi, Tamamori and Toda47], because of the mismatches of speech features, such as the oversmoothed characteristics of the estimated spectral features due to the mean-based optimization of a spectral conversion model. To improve the quality of the converted speech waveform, these mismatches between the naturally extracted speech parameters and the oversmoothed estimated speech parameters have to be alleviated, e.g. with a post-conversion processing method.

The ideal way to address the degradation problem of WN vocoder in VC is by directly addressing the mismatches of speech features in the development of a WN vocoder. In this work, as a first step toward improving the use of the neural vocoder in VC, we propose to overcome the quality degradation by the use of post-conversion processing to alleviate the feature mismatches, such as oversmoothing problem. The proposed post-conversion processing is based on GV [Reference Toda, Black and Tokuda22] postfilter and DiffVC [Reference Kobayashi, Toda and Nakamura5], which is illustrated in Fig. 3. The GV postfilter is used to recover the variances of the oversmoothed spectral features to make them closer to the natural ones. The DiffVC method is used to further reduce the mismatches between the spectral and excitation features by directly modifying the input speech waveform according to the differences between the estimated target spectra and the input spectra. Then, through the combination of GV and DiffVC post-conversion processing (DiffGV), the modified set of speech parameters [Reference Tobing, Wu, Hayashi, Kobayashi and Toda24] can be obtained for use in the WN vocoder as shown in Fig. 4. Note that, as shown in Fig. 3, for cross-gender conversions, e.g. male-to-female speakers, an additional step of analysis–synthesis is performed with also the transformed excitation features because the DiffVC [Reference Kobayashi, Toda and Nakamura5] method does not modify the original excitation when filtering to generate the modified waveform using the mel-log spectrum approximation (MLSA) filter [Reference Imai, Sumita and Furuichi55].

Fig. 3. Flow of direct waveform modification with spectrum differential (DiffVC) [Reference Kobayashi, Toda and Nakamura5] using MLSA synthesis filter [Reference Imai, Sumita and Furuichi55] and GV postfilter [Reference Toda, Black and Tokuda22].

Fig. 4. Waveform generation procedure for converted speech using WN vocoder through the use of post-conversion processed auxiliary speech features based on the DiffVC method and the GV postfilter (DiffGV).

Finally, to preserve the consistency between the spectral conversion model and the waveform generation procedure, due to the use of spectrum differential method (DiffVC), we propose the use of an alternative loss function for the RNN-based architectures that considers the spectrum differences. Given the estimated feature vector $\hat {\boldsymbol {y}}_t=f_{\boldsymbol {\lambda }_\textrm {R}}(\boldsymbol {x}_t)$ from (8), instead of optimizing the parameters with respect to the ground truth $\boldsymbol {y}_t$, the optimization is performed as follows:

(21)\begin{align} \hat{\boldsymbol{\lambda}}_\textrm{R}=\arg\,\min_{\boldsymbol{\lambda}_\textrm{R}}\frac{1}{T}\sum_{t=1}^{T} \frac{10\sqrt{2}}{\ln{10}}\sum_{d=1}^{D}|\hat{y}_t(d)-(y_t(d)-x_t(d))|, \end{align}

where $f_{\boldsymbol {\lambda }_\textrm {R}}(\cdot )$ is the feedforward function using the RNN-based spectral conversion model, as described in Section 3.3, and $|\cdot |$ denotes the absolute function. Therefore, the conversion model is optimized to directly generate the spectrum differential trajectory, i.e. the trajectory difference between the target $\boldsymbol {y}$ and the source spectra $\boldsymbol {x}$, which is used to generate the set of post-conversion processed speech features for generating the converted speech waveform with the WN vocoder. As a further note, the constant coefficient in (15) and (21) comes from the MCD formulation [Reference Mashimo, Toda, Shikano and Campbell54], which is quite effective in our experimental conditions on the RNN-based spectral modeling.

A) Experimental conditions

In the experiments, we used the VCC 2018 dataset [Reference Lorenzo-Trueba51], consisting of six female and six male speakers, as well as additional data from the CMU Arctic dataset [Reference Kominek and Black56], where we used the data of “bdl” (male) and “slt” (female) speakers to develop a multispeaker WN vocoder [Reference Hayashi, Tamamori, Kobayashi, Takeda and Toda45]. On the other hand, to develop the spectral conversion models, we used a subset of the VCC 2018 data, comprising “SF1” and “SM1” data as the source speakers, as well as “TF1” and “TM1” data as the target speakers, where “F” means female and “M” means male, giving a total of four conversion models for each of the network architectures described in Section III. The total numbers of utterances in the CMU Arctic dataset and VCC 2018 dataset are 1132 and 81, respectively. The training set from the CMU Arctic utterances consisted of the first 992 utterances, whereas that from the VCC 2018 dataset consisted of the final 71 utterances. The remaining sentences in both datasets were used as testing data. The number of evaluation utterances provided in the VCC 2018 dataset is 35, which were used in the subjective evaluation. The length of each audio sample in the training data was roughly 3.5 s on average.

As the spectral features, we used the 34D mel-cepstrum parameters, including the 0th power coefficient, extracted from the spectral envelope of the WORLD spectrum [Reference Morise, Yokomori and Ozawa38, Reference Hsu57]. The frequency warping parameter was set to $0.455$ [Reference Yamamoto58]. The spectral envelope of the speech spectrum was computed frame by frame using CheapTrick [Reference Morise59, Reference Morise60] and then parameterized into the mel-cepstrum coefficients [Reference Yamamoto58]. We used framewise $F_0$ values as the excitation features as well as the two-band aperiodicity features, which were, respectively, extracted using Harvest [Reference Morise61] and D4C [Reference Morise62] in the WORLD package. To perform the prosody conversion, we carried out a linear transformation of the $F_0$ values using the statistics of the source and target speakers. For the set of auxiliary speech parameters used in the WN vocoder, we utilized continuous interpolated $F_0$ values and binary unvoiced/voiced (U/V) decisions, giving 39D auxiliary speech parameters, i.e. 1D U/V, 1D continuous F0, 2D code-aperiodicity, 0th power, and 34D mel-cepstrum. The speech signal sampling rate was 22,050 kHz and the frameshift was set to $5$ ms.

The WN architecture comprised a $1,2,4,\dotsc ,1024$ dilation sequence with four repetitions. The number of residual channels was 128 and the number of skip channels was 256. Two convolution layers with ReLU activation were used after the skip connections before the softmax output layer. The trained multispeaker WN model was fine-tuned for each target speaker, i.e. TF1 or TM1, using their extracted speech parameters. The implementation of WN was based on [Reference Hayashi63], where the noise-shaping method [Reference Tachibana, Toda, Shiga and Kawai64] was also used. To train the WN model parameters, the Adam algorithm [Reference Kingma and Ba65] was employed. The weights of model parameters were initialized using Xavier [Reference Glorot and Bengio66], while the biases were zero-initialized. The learning rate was set to 0.0001. The hyperparameters for WN training, i.e. the optimization algorithm, the initialization method, and the learning rate, were set to the same as those for the training of spectral conversion models.

In the development of the spectral conversion models, we used five hidden layers for the DNN model and four hidden layers for the DMDN model with ReLU activation functions. The number of mixture components for the DMDN was set to 16. For the RNN-based models, we used one layer for both hidden LSTM/GRU units. Convolutional input layers were used for all DNN-/DMDN-/RNN-based spectral models, where they were designed to capture four preceding and four succeeding frame input contexts with dynamic dimensions. Specifically, two convolutional input layers were used with a kernel size of 3 and a dilation size of 1 and 3, respectively, for each layer. The number of output dimensions for each layer was its kernel size multiplied by the number of input dimensions, e.g. with 50D features, the first input layer will have 150D output and the second layer will have 450D output. Dropout [Reference Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov67] layers with 0.5 probability were used in the training of the RNN-based models after the convolutional input layers and after the hidden LSTM/GRU units. For the RNN-based methods, we also trained additional models using the proposed loss for the spectrum differential in (21). An additional normalization layer before the input convolutional layers and a de-normalization layer after the final output layer were used and fixed according to the statistics obtained from the training data. The performance of all conversion models was compared in the objective evaluation. In the subjective evaluation, we used the GRU-based conversion model with both the conventional loss in (15) and the proposed spectrum differential loss in (21). The dynamic-time-warping procedure was used to create time-warping functions for computing the loss between the estimated target spectra and the ground truth by only using the speech frames (non-silent frames).

B) Objective evaluation

To objectively evaluate the NN-based spectral mapping models described in Section III, we computed the metrics of MCD [Reference Toda, Black and Tokuda22, Reference Mashimo, Toda, Shikano and Campbell54] and LGD. These metrics were computed between the converted mel-cepstrum parameters of the source speaker and the extracted (natural) mel-cepstrum parameters of the target speaker for all speaker pairs, i.e. SF1–TF1, SF1–TM1, SM1–TF1, and SM1–TM1.

The MCD was computed using

(22)\begin{equation} \text{MCD} [\text{dB}] = \frac{1}{T}\sum_{t=1}^{T}\frac{10}{\ln{10}}\sqrt{2\sum_{d=2}^{D}(\hat{y}_t(d)-y_t(d))^2}, \end{equation}

where $\hat {y}_t(d)$ is the $d$th dimension of the converted mel-cepstrum and $y_t(d)$ is that of the target mel-cepstrum at frame $t$. The starting dimension $d$ is set to 2 for only measuring the mel-cepstrum parameters without the 0th power. On the other hand, to compute the LGD, the following function was used:

(23)\begin{equation} \text{LGD}=\frac{1}{D}\sum_{d=2}^{D}\sqrt{(\log{\sigma_{\text{global}}(\hat{y}(d))}-\log{\sigma_{\text{global}}(y(d))})^2}, \end{equation}

where the GV [Reference Toda, Black and Tokuda22] $\sigma _{\text {global}}(\cdot )$ is computed as

(24)\begin{equation} \sigma_{\text{global}}(y(d)) = \frac{1}{N}\sum_{n=1}^{N}\frac{1}{T_n}\sum_{t=1}^{T_n}(y_t(d)-\overline{y}_n(d))^2. \end{equation}

The index of the $n$th training data is denoted by $n$, the total number of training sentences is $N$, and the mean value of the $d$th dimension target spectra of the $n$th utterance is denoted as $\overline {y}_n(d)$. The objective measurements of all the spectral conversion models were computed. These models consisted of the DNN models, the DMDN models, the LSTM-based models, the GRU-based models with conventional loss, and the GRU-based models with the proposed spectrum differential loss in (21) (GRUDiff).

The results for MCD measurements of the training and testing data, averaged over all speaker-pair conversions, are shown in Figs 5 and 6, respectively, during 500 training epochs for DNN/DMDN models, and during 325 epochs for LSTM/GRU/GRUDiff models. The lower number of epochs for RNN-based models is due to their faster convergence rate compared to the conventional DNN-/DMDN-based models. It can be observed that the GRU-based spectral models with conventional loss give better accuracy and generalization for unseen data, where it yields higher accuracy than the other models within only 50–80 epochs. Similar tendencies are also observed for the same- and cross-gender conversions, as, respectively, shown in Figs 7 and 8. Note that the lower accuracy for the GRUDiff models is due to the optimization for generating the spectrum differential rather than the target spectra. Although the MCD measurement does not always correlate with perceptual output, they can still be used as a basic metric for monitoring the convergence and the basic performance of spectral mapping models. As a further side note, for the training time, one epoch of the DNN/DMDN-based network takes about two times faster than the LSTM-/GRU-based network. However, the convergence of the LSTM-/GRU-based network takes only about 50–80 epochs, whereas that of the DNN/DMDN takes nearly 500 epochs.

Fig. 5. Trend of mel-cepstral distortion (MCD) for the training set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

Fig. 6. Trend of MCD for the testing set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

Fig. 7. Trend of MCD for same-gender (SG) conversions on the testing set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

Fig. 8. Trend of MCD for cross-gender (XG) conversions on the testing set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

Hence, to accompany the pure accuracy results of MCD, the LGD measurements were performed to evaluate the oversmoothness (variance reduction) of spectral trajectory due to the mean-based optimization in spectral modeling. The results of the LGD measurements of the testing data for same- and cross-gender conversions are given in Figs 9 and 10, respectively. These results demonstrate that the proposed spectrum differential loss used for the GRUDiff models makes it possible for the resulting converted trajectory to be less oversmoothed. Furthermore, all of the RNN-based spectral models show a clear tendency of being capable of producing better trajectory variance compared to the DNN-/DMDN-based models, such as the one used in our previous work [Reference Tobing, Wu, Hayashi, Kobayashi and Toda24], i.e. DMDN-based model. Considering the capability of GRU and GRUDiff to produce superior objective results, we used them both to generate converted spectra in the subjective evaluation. As to the latter point, in this work, we indeed focus on providing empirical evidence of the performance of RNN-based architectures for the VC framework, which would be beneficial for the continuous future development of real-world applications and integration of training development with other frameworks, such as neural vocoder.

Fig. 9. Trend of log-GV distance (LGD) for SG conversions on the testing set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

Fig. 10. Trend of LGD for XG conversions on the testing set using the DNN-, DMDN-, LSTM-, GRU-, and GRUDiff-based spectral conversion models during 500 training epochs for the DNN/DMDN models and 325 training epochs for the LSTM/GRU/GRUDiff models.

C) Subjective evaluation

In the subjective evaluation, we conducted MOS tests to evaluate the naturalness of the converted speech waveforms and speaker similarity tests to evaluate the accuracy of the converted speech waveforms. In the MOS tests, a five-scale score was used to assess the naturalness of speech utterances, i.e. 1: completely unnatural, 2: mostly unnatural, 3: equally natural and unnatural, 4: mostly natural, 5: completely natural. On the other hand, for the speaker similarity tests, each listener was given a pair of stimuli, i.e. a natural speech of the target speaker and a converted speech of the source speaker, and asked to judge whether or not the two speech utterances were produced by the same speaker. To judge the similarity, each listener chose between two main responses, i.e. “same” or “different”, where each had a confidence measurement, i.e. “sure” or “not sure”, giving a total of four options when judging the similarity. The number of listeners was 10.

We compared the use of direct waveform modification with the spectrum differential, i.e. DiffVC [Reference Kobayashi, Toda and Nakamura5], using the GV [Reference Toda, Black and Tokuda22] postfilter (dG), which was made to nearly matching the baseline system of VCC 2018 [Reference Kobayashi and Toda68], and the use of the WN vocoder to generate the converted speech waveforms. In the case of the WN-based waveform generation, we also compared the use of several types of speech auxiliary features, namely, converted spectral features (WNc), converted features with the GV postfilter (WNcG), post-conversion processing based on DiffVC (WNd), and DiffVC post-conversion processing with the GV postfilter (WNdG), which was used for our VC system in VCC 2018 [Reference Tobing, Wu, Hayashi, Kobayashi and Toda24]. Furthermore, we also utilized two different spectral mapping models, i.e. the GRU-based models with conventional loss (GRU) and spectrum differential loss (GRUDiff). The total combinations of speaker pairs and spectral mapping models/waveform generation methods were 40, i.e. four speaker pairs and ten different models/methods. In the naturalness evaluation, the number of distinct utterances was three, whereas, in the speaker similarity evaluation, it was two. In both evaluations, we also included the original speech waveform of the four speakers, i.e. either as a single stimulus of the original waveform in naturalness test or a pair of stimuli of original waveforms (from either source–target/target–target) in similarity test. Therefore, each listener had to evaluate 132 audio samples in the naturalness test and 92 audio samples in the similarity test.

The results of the MOS tests are shown in Tables 1 and 2. It can be observed the DiffVC with the GV method yields relatively high naturalness scores for same-gender conversions but not for cross-gender conversions owing to the use of the conventional vocoder in the cross-gender conversions. On the other hand, for the WN-based waveform generation, the use of post-conversion processing based on DiffVC and the GV, i.e. WNdG, clearly enhances the naturalness of the converted speech waveforms compared with the other sets of auxiliary speech features. Furthermore, the proposed spectrum differential loss improves the performance for same-gender conversions, i.e. either F-to-F or M-to-M.

Table 1. Results of mean opinion score (MOS) test of DiffVC with the GV postfilter (dG) waveform generation method using either GRU or GRUDiff spectral mapping models and of the original speech signals.

$\pm$ denotes the $95\%$ confidence interval. S-Gender and X-Gender denote same-gender and cross-gender conversions, respectively.

Table 2. Results of MOS test of the WN-based generation methods using plain converted mel-cepstrum (c), using c with GV postfilter (cG), using post-conversion based on DiffVC (d), and using d with GV postfilter (dG) from either GRU or GRUDiff spectral mappings.

S-Gender and X-Gender, respectively, denote same-gender and cross-gender conversions. $\pm$ denotes the $95\%$ confidence interval of the sample mean. Bold indicates the system(s) with the highest mean score in each conversion category. [$\cdot$] Denotes a system with a statistically significant lower score than the highest score in each conversion category. Statistical inferences were performed using the two-tailed Mann–Whitney test with $\alpha <0.05$.

Results for the speaker similarity tests are given in Table 3. The results show that the use of the WN vocoder with the DiffGV-based post-conversion processed auxiliary features (WNdG) yields superior accuracy to the other methods, including the conventional DiffGV without the WN vocoder (dG). The results for naturalness and accuracy in this paper exhibit a similar tendency to the results of VCC 2018 [Reference Lorenzo-Trueba51], where the baseline method [Reference Kobayashi and Toda68] yields better naturalness, especially for same-gender conversions, due to the avoidance of the conventional vocoder, and our VC system with post-conversion processing for the WN vocoder [Reference Tobing, Wu, Hayashi, Kobayashi and Toda24] gives better accuracy. Based on these results, for the use of a WN vocoder in VC, it has been shown that an additional processing procedure adjusted to reduce the mismatches between estimated and natural speech features is needed to improve the converted speech waveform. For future work, it is worthwhile to directly address this problem in the development of a neural vocoder, such as WN, for VC, i.e. in a data-driven manner. All audio samples used in the subjective evaluation are available at http://bit.ly/2WjuJd1.

Table 3. Results of speaker similarity test (scores were aggregations of “same – sure” and “same – not sure” decisions) of the converted speech waveform using all waveform generation methods (dG, WNc, WNcG, WNd, and WNdG) with either GRU or GRUDiff spectral mappings.

S-Gender and X-Gender, respectively, denotes same-gender and cross-gender conversions. Bold indicates the system(s) with the best similarity score in each conversion category. [$\cdot$] Denotes a system with a statistically significant lower score than the best score in each conversion category. Statistical inferences were performed using the two-tailed Mann–Whitney test with $\alpha <0.05$.