A) Visualizing the speaker variability in the feature space

Initially, we visualize the distribution of features from both genuine and spoofed speech corresponding to multiple speakers by projecting the feature space onto a 2-D plane (refer Fig. 2) via t-SNE. Based on this, we can observe distinct clustering of the features corresponding to different speakers in both genuine and spoofed speech indicating that speaker variability is a significant factor in both. Moreover, it is interesting to observe that the speaker clusters are somewhat more distinct in genuine speech features than in spoofed speech features suggesting that genuine speech models may be more affected by speaker variability.

Fig. 2. Feature space (STMF) projected onto 2-D via t-SNE indicates the presence of clusters corresponding to speakers in both genuine and spoofed speech from the ASVspoof2017 v2.0 corpus. The same markers are used for corresponding speakers in genuine and spoofed classes.

The above visualization just shows a very abstract idea about speaker variability. To quantify the speaker variability, we should analyze the differences in probabilistic distributions over the feature space, which captures the differences in the acoustic characteristics of different speakers in genuine and spoofed classes.

B) Quantifying the speaker variability

While the above visualization (Fig. 2) provides a clear indication that speaker variability affects both genuine and spoofed speech, it makes no attempt to quantify the degree of variability. In order to do so, we now model the underlying feature distributions for each speaker and estimate the differences between these distributions. Specifically, given the joint distribution $P(x,y|spk)$ of features $x$ and class $y$ for each speaker $spk$, it can be expected that the differences in the speech characteristics between two speakers, $spk = i$ and $spk = j$ will correspond to the differences between the distributions ${{{\mathcal P}}_i} = P({x|spk = i})$ and ${{{\mathcal P}}_j} = P(x|spk = j)$. For example, if speaker variability is a significant confounding factor for a model, we would expect that the differences between the distributions corresponding to any two target speakers to be greater than the differences between the distribution of a target speaker, $P(x|spk)$ and the SI distribution, $P(x)$.

In this work, the differences between distributions are estimated as the Kullback–Leibler (KL) divergence between the corresponding GMMs. KL divergence is generally used to measure the distance between two probabilistic models, $({{{\mathcal P}}_1},{{{\mathcal P}}_2})$. Given a D-dimensional feature vector, $X \epsilon {{{\mathbb R}}^D}$, the KL divergence is of ${{{\mathcal P}}_2}$ from ${{{\mathcal P}}_1}$ is defined as [Reference Kullback33]:

(1)\begin{equation}KL({{{{\mathcal P}}_1},{{{\mathcal P}}_2}}) = \int_X {{{\mathcal P}}_1}(X) \ln \left( {\frac{{{{{\mathcal P}}_1}(X) }}{{{{{\mathcal P}}_2}(X) }}} \right)\textrm{d}X\end{equation}

As $KL({{{{\mathcal P}}_1},{{{\mathcal P}}_2}})$ is an asymmetric divergence measure, i.e., $KL({{{{\mathcal P}}_1},{{{\mathcal P}}_2}}) \; \ne \; KL({{{{\mathcal P}}_2},{{{\mathcal P}}_1}})$, a symmetric KL divergence, ${S_{KL}}$, is defined as [Reference Kullback33]:

(2)\begin{equation}{S_{KL}}({{{{\mathcal P}}_1},{{{\mathcal P}}_2}}) = \frac{1}{2}({KL({{{{\mathcal P}}_1},{{{\mathcal P}}_2}}) + KL({{{{\mathcal P}}_2},{{{\mathcal P}}_1}}) }) \; \end{equation}

A Monte Carlo approximation-based symmetric KL divergence [Reference Sethu, Epps and Ambikairajah34] is used to measure the distance between two probability distributions. Average inter-speaker KL divergence (between a target speaker's distribution and all other target speakers' distributions) of a speaker is estimated to measure the separation between speaker distributions, $P(x|spk)$. The average inter-speaker KL divergence, ${I_{KL}}$, for each speaker, $i$, is given as follows:

(3)\begin{equation}{I_{KL}}(i) = \frac{1}{{{N_{spk}} - 1}}\mathop \sum \limits_{j = 1}^{{N_{spk}}} {S_{KL}}({{{{\mathcal P}}_i},{{{\mathcal P}}_j}}) ;\quad i \ne j\end{equation}

where ${N_{spk}}$ is the total number of claimed speakers. ${S_{KL}}({{{{\mathcal P}}_i},{{{\mathcal P}}_j}})$ is the KL divergence between ${i^{th}}$ and ${j^{th}}$ speaker distributions as shown in equation 2. In addition to that, the KL divergence between a target speaker's feature distribution and the SI distribution, ${U_{KL}}$, is then given by:

(4)\begin{equation}{U_{KL}}(i) = {S_{KL}}({P({x\vert spk = i}) , P(x) }) \end{equation}

where $i$ is a speaker and $P(x)$ is the SI distribution and $P({x\textrm{|}spk = i})$ is ${i^{th}}$ speaker distribution.

All the feature distributions are modeled as four mixture GMMs of the STMF space for this analysis. Target speaker's feature distributions are then estimated via Maximum a Posteriori (MAP) adaptation from SI distributions. Genuine and spoofed speaker distributions and SI distributions are trained separately with genuine and spoofed data, respectively.

The average inter-speaker KL divergences (${I_{KL}}$ as in equation (3)) and the KL divergence between a target speaker's feature distribution and the SI distribution (${U_{KL}}$ as in equation (4)), for multiple speakers, are compared in Fig. 3 for both genuine and spoofed classes, separately. Additionally, the mean values of both measures across all considered speakers also drawn in Fig. 3. This comparison was carried out on the ASVspoof2017 v2.0 speaker-specific enrolment set (refer to section IV for details about the database and the enrolment set). From Fig. 3, it can be seen that the average inter-speaker KL divergence between one speaker and all other speakers $({I_{KL}})$ is greater than the one between that speaker and SI distributions $({U_{KL}})$, for both genuine and spoofed classes, which indicates that there is clear separation between probability distributions corresponding to different target speakers.

Fig. 3. Symmetric KL divergence for speaker-dependent distributions and SI distributions of probability distributions for all speakers in (a) genuine class and (b) spoofed class. This is used to quantify the difference between different speakers as well as the difference between speaker-specific and SI models (speaker variability). The STMF are used as the front-end in this analysis carried out on the ASVspoof2017 v2.0 corpus.

Also, genuine and spoofed classes showed different degrees of speaker variability. Specifically, (1) KL between genuine speaker distribution versus genuine SI distribution is higher than spoofed speaker distribution versus spoofed SI distribution for each corresponding speaker (refer Figs. 3(a) and 3(b), ${U_{KL}}$‘s and mean of ${U_{KL}}$); (2) inter-speaker KL divergences of genuine classes are greater than inter-speaker KL divergences of spoofed classes (refer Figs. 3(a) and (b), ${I_{KL}}$‘s and mean of ${I_{KL}})$. Thus, the implication of this observation can be interpreted as spoofed class containing less speaker variability than genuine which is in line with our feature space visualization.

C) Analysis of speaker-specific information in the features

In addition to quantifying the degree of speaker variability based on models of  the feature distributions of target speakers, we also quantify it in terms of how well speaker identification can be carried out. Specifically, we build simple GMM-based speaker identification systems on both genuine speech as well as replayed speech and use the speaker identification accuracy as an indicator of the level of speaker variability.

The speaker identification systems, for both genuine and replayed speech, were set up as 512 and 4 mixture GMM-universal background model (UBM) systems using both CQCC and STMF front-ends, respectively. The accuracies of these systems are reported in terms of identification rates (ratio of number correctly identified to total number of test utterances) and estimated on held out test sets comprising 20 utterances per speaker each, for genuine speech and spoofed (replayed) speech. There were a total of 15 speakers in the test set, which corresponds to a chance accuracy of ~6.6%. From the results reported in Table 1, it can be seen that for both genuine speech and replayed speech, the speaker identification rate is significantly higher than chance indicating that a high level of speaker variability is present in the features. It is also interesting to note that the speaker identification rate on genuine speech is much higher than replayed speech.

Table 1. Speaker identification accuracies on genuine speech and replayed speech evaluated on ASVspoof2017 v2.0 using a GMM-UBM speaker identification system.

D) Analysis of model separation: genuine versus spoofed classes

Even though the analyses in the previous three sections strongly suggest that both genuine and spoofed speech is greatly affected by speaker variability, it does not directly imply that the use of target speakers' genuine and spoof distributions will lead to better spoofing detection. In order to discern this, we now quantify the ability to discriminate between genuine and spoofed speech with both target speaker's distributions and SI distributions.

It should be noted that a set of GMMs that perform well as a classifier will have a large degree of mutual dissimilarity and consequently a large KL value when compared with a set of GMMs that are more similar to each other. To analyze the discriminability three KL divergence measures are estimated. (1) KL divergence between genuine SI distribution (genuine SI) and spoofed SI distribution (spoofed SI),$\; D{U_{KL}}$, is given by:

(5)\begin{equation}D{U_{KL}} = {S_{KL}}({P({x\vert y = G}) , P({x\vert y = S}) }) \end{equation}

where features $x$ and class $y$ with two classes genuine,$\; G$ and spoofed, $S$; (2) the spoofed SI and each of the genuine target speaker's distributions (genuine speaker models), $D{1_{KL}}$, asfollows:

(6)\begin{equation}D{1_{KL}}(i) = {S_{KL}}({P({x\vert y = G,spk = i}) , P({x\vert y = S}) }) \end{equation}

where features $x$ and class $y$ of genuine,$\; G$ for ${i^{th}}$ speaker, $spk$ and spoofed, $S$; (3) each of the genuine speaker models and each of the spoofed speaker models, $D{2_{KL}}$, is given by:

(7)\begin{equation}D{2_{KL}}(i) = {S_{KL}}({P({x\vert y = G,spk = i}) , P({x\vert y = S,spk = i}) }) \end{equation}

where features $x$ and class $y$ of genuine,$\; G$ and spoofed, $S$ for ${i^{th}}$ speaker, $spk$. These three KL measures, such as $D{U_{KL}}$, $D{1_{KL}}$, and $D{2_{KL}}$ are compared in Fig. 4 for two different features, such as CQCC and STMF.

Fig. 4. Comparison of KL divergence between genuine and spoofed models of: (a) STMF and (b) CQCC features for both SI distributions and speaker-dependent distributions on the ASVspoof2017 v2.0 corpus. This is used to quantify the discriminability between genuine and spoofed speech (for each speaker) and compare between speaker-specific and speaker-independent models.

As observed in Fig. 4, both $D{1_{KL}}$ and $D{2_{KL}}$ constantly show higher KL than $D{U_{KL}}$ for both CQCC and STMF features. Also the difference between $D{2_{KL}}$ and $D{1_{KL}}$ is not considerably higher than that between $D{U_{KL}}$ and $D{1_{KL}}$. This might be due to the scarcity of speaker-related information in the spoofed class over the genuine class which is in line with the observation from Fig. 3. Taken together, the results shown in Table 1 and Fig. 4, we can notice that short-term CQCC have rich speaker-related information than long-term STMF.

The analysis of this section concurs with the observations made in the case of feature distributions and further support that the speaker variability is a significant confounding factor and make use of these speaker-dependent distributions for spoofing detection will be beneficial. Thus, we proposed a framework based on the fact that the use of claimed speaker genuine and/or spoofed models extinguishes speaker variability and leads to better discrimination between genuine and spoofed classes.

A) ASVspoof2017 (V2.0) corpus

The evaluation partition of the ASVspoof2017 v2.0 corpora comprises genuine speech corresponding to multiple utterances of 10 passphrases from 24 speakers (same 10 passphrases across all speakers). In addition, replayed versions of these utterances under 57 different replay conditions (combination of recording device, environmental acoustic channel, and playback device) from 17 of the 24 speakers are also part of the evaluation partition. In addition, these 57 RCs are marginalized as low, medium, and high threats in terms of recording, environment, and playback conditions [Reference Todisco, Evans, Kinnunen, Lee and Yamagishi24].

As noted in Fig. 8, replayed speech was not available for seven speakers. Consequently, we only use the remaining 17 speakers in our evaluation. When repartitioning this evaluation set (of 17 speakers) into enrolment and test sets, we select one utterance corresponding to each passphrase as the “genuine enrolment set.” The remaining genuine utterances are incorporated into the held out “test set.”

Fig. 8. Number of utterances for each speaker in the ASVspoof2017 v2.0 evaluation set. The speaker IDs provided in the dataset are indicated along the x-axis.

Of the 57 RCs present in the evaluation partition, only seven overlap with the train and dev sets. Consequently, we include the spoofed data corresponding to these seven replay conditions into the “spoofed enrolment set” and retain the data corresponding to the remaining 50 unseen RCs (unseen in terms of training the spoofing detection models) in the held-out, “test set.” The number of utterances in the enrolment and test sets is highlighted in Table 2.

Table 2. ASVspoof2017 Version 2.0 [Reference Todisco, Evans, Kinnunen, Lee and Yamagishi24]

*The details of “Enrolment set” and the “Speaker-specific Test set” can be found in http://www2.ee.unsw.edu.au/ASVspoof/.

B) BTAS 2016 corpus

BTAS 2016, a text-independent database, contains genuine and different kinds of spoofing attacks where genuine, speech synthesis and voice conversion speech samples were recorded and played back using high-quality devices [Reference Korshunov36]. We have only used “replay subset” of BTAS for our experiments which comprises replayed versions of the genuine speech utterances as in [Reference Xie, Zhang, Chen and Xu37]. The number of utterances for each of the train, development and evaluation partitions is presented in Table 3 and information about speaker data given in Fig. 9. Unknown RCs are present in the evaluation partition to make the spoofing detection task more challenging.

Table 3. “Replay Subset” of the BTAS2016 corpus.

*The details of “Enrolment set” and the “Speaker-specific Test set” can be found in http://www2.ee.unsw.edu.au/ASVspoof/.

Fig. 9. Number of utterances from each speaker in the BTAS 2016 evaluation set. The speaker IDs provided in the dataset are indicated along the x-axis.

Similar to ASVspoof2017 v2.0, we partition the evaluation set of the BTAS corpus into an enrolment set and a test set by taking 30 genuine and 30 spoofed utterances from every speaker to form the enrolment set (rest constitute the “test set’). The data statistics of both data sets are summarized in Table 3. Once again, the enrolment set only contains known RCs (all unknown RCs are in the test set) and held out “test set” kept as the same characteristics (in terms of replay conditions) as the original BTAS 2016 evaluation set.

A) Front-end feature configurations

Prior to all feature extraction, the speech was pre-emphasized with 0.97 factor. It should be noted that no normalization is applied to any of the features in this work.

STMF parameters: Pre-emphasized speech is framed with a 50% overlap between them using a Hamming window. The STMF features are extracted using the same parameters and MCF-CC and MSE-CC features are chosen with 15 and 30 dimensions respectively and feature-level concatenation is performed to obtain 45 dimensions for each utterance as in [Reference Suthokumar, Sethu, Wijenayake and Ambikairajah22].

LMGD parameters: LMGD features are extracted from frames of 50% overlap followed by Hamming window. $\alpha, \gamma$ were empirically chosen to be 0.4 and 0.9, respectively, based on the development set. LMGD feature is 120 dimensions which consists of static, velocity, and acceleration coefficients.

CQCC parameters: For the derivation of CQCC features, we have used the same configuration as used in the ASVspoof2017 challenge baseline [Reference Kinnunen2]. A Constant-Q Transform is applied with a maximum frequency of ${f_{\textrm{max}}} = 8\;\textrm{kHz}$, which is the Nyquist frequency and the minimum frequency, ${f_{\textrm{min}}} = {f_{\textrm{max}}}/{2^9} \approx 15\,\textrm{Hz}$, where 9 is the number of octaves. The number of bins per octave is set to $96$. Resampling is applied with a sampling period of 16 bins in the first octave. The CQCC feature dimension is set to 90 coefficients.

B) GMM backend configuration

In the proposed GMM system, the spoofed and genuine UBMs were both implemented as 512 mixture GMMs for CQCC and LMGD features and as 4 mixture GMMs for STMF features. All UBMs were trained using the EM algorithm with random initialization. In the experiments carried out on the ASVspoof2017 v2.0 corpus, the Train and Dev sets were used to model the genuine and spoofed UBMs (no overlap with test speaker data). For the system evaluated on BTAS 2016, the train set was used to model the UBMs. The speaker-dependent genuine and spoofed models were then estimated from the UBMs via MAP adaptation (mean and weights only) using the Enrolment set (refer section V for details on the enrolment set).

C) DNN backend configuration

The SI baseline systems, using either the CQCC or the LMGD front-ends, were trained using Train and Dev sets for the ASVspoof2017 v2.0 system and the Train set for the BTAS 2016 system. Two variations of this DNN system were developed and evaluated, one with five fully-connected (FC) layers and the other with three residual network layers sandwiched between two FC layers as shown in Fig. 12. In both systems, the final FC layer has 64 neurons and the first FC layer has as many neurons as the input feature dimension. The middle three FC layers of the first system (Fig. 12(a)) have 1024, 512, and 256 neurons, respectively (going from the input to output). The three residual layer blocks of the second system (Fig. 12(b)) all have an identical architecture with each block comprising two layers of weights comprising 256 and 128 neurons each. All hidden layer neurons employ a ReLU activation function and the final layer uses softmax activation. Dropouts and batch normalization were employed during model training, using a cross-entropy loss function, and the dropout rate was set as 0.4. Learning rates were decayed uniformly across the network per epoch from ${10^{ - 2}}$ to ${10^{ - 5}}$. Early stopping based on validation loss on the dev set was used to avoid overfitting. All network weights are initialized from a normal distribution with zero-mean and a standard deviation of ${10^{ - 2}}$. Layerwise L2 regularization was also performed.

Fig. 12. Two variants of the DNN that can be employed as the SI and speaker-dependent DNNs in Fig. 6: (a) five FC layers; and (b) three residual layers in between two FC layers.

Finally, to obtain speaker-dependent DNN back-ends for our proposed system, the trained (SI models were used as initial models and the final two layers were retrained for three additional epochs using the enrolment data from the target speaker. When using CQCC and LMGD features, the average posterior predicted by the back-end over all the frames in an utterance is used as the score to calculate EERs.

The DNN back-end Architecture chosen for the STMF front-end were constrained to be a smaller version of that used by the CQCC and LMGD systems to avoid over parameterization. Since STMF is an utterance-based feature, a single set of posteriors is predicted by the back-end and used as the score to estimate EERs.