A) Noise suppression technology

When a call partner is in a noisy environment, it may become difficult for the user to hear the received voice due to the surrounding noise. Therefore, noise suppression technologies for suppressing only the noise from the received voice, with which the voice and the noise mix, to improve voice quality have been developed.

The noise suppression technologies are classified into the method with multiple microphones, and the method with single microphone. As a fundamental technology for suppressing the noise with single microphone, spectral subtraction method was proposed [Reference Boll2]. Fig. 1 is the block diagram of spectral subtraction method. With this method, noise components are suppressed by deducting the estimated noise spectrum from the input spectrum in a frequency domain. However, the noise components may remain because of the error of the estimated noise spectrum, and residual noise appears and disappears, as a result, it is easy to cause unpleasant sound called musical noise.

Fig. 1.

Block diagram of spectral subtraction method.

As a technology for maximizing signal-to-noise ratio (SNR) that is power ratio of voice and noise and improving amount of noise suppression, minimum mean-square-error short-time spectral amplitude (MMSE-STSA) method was proposed [Reference Ephraim and Malah3]. Fig. 2 is the block diagram of MMSE-STSA method. In this method, the voice spectrum is obtained by applying the gain to the input spectrum. The gain is calculated by minimizing the mean square error between the voice spectrum obtained from input and estimated noise and the voice spectrum obtained by applying the gain to input spectrum in the condition of assuming the independence of the voice and the noise. The error of the voice spectrum can be evaluated by using a posteriori SNR (the ratio of the estimated voice power and the estimated noise power) and a priori SNR (the ratio of the input signal power to the estimated noise power). With this method, the amount of noise suppression becomes higher than those of above spectral subtraction method because of maximizing the power ratio of voice and noise. In addition, the power of the musical noise caused by this method is lower because of restoring the voice without deducting the estimated noise.

Fig. 2.

Block diagram of MMSE-STSA method.

B) Reverberation suppression technology

When a call partner is in a narrow room, it may become difficult for user to hear the received voice because the partner's voice mixes with the reverberant sound that is sound to which the partner's voice reflects in the walls and ceiling. Therefore, reverberation suppression technologies for suppressing only the reverberant sound from the received voice, with which the voice and the reverberant sound mix, for improving voice quality have been proposed.

The reverberation suppression method that uses multi-step linear prediction in the time-domain has been proposed [Reference Kinoshita, Nakatani and Miyoshi4]. The effect of decreasing the reverberation time and that of improving the voice recognition rate by reverberation suppression are shown. Fig. 3 is the block diagram of the reverberation suppression method. With this method, the reverberation components are estimated from the reverberant speech signal by the linear prediction, and the reverberation components are suppressed using the spectral subtraction method. This method has been put to practical use as a plug-in feature for audio editing software [Reference Kinoshita5].

Fig. 3.

Block diagram of the reverberation suppression method.

C) Voice enhancement technology for audio CODEC

About the voice enhancement for audio CODEC, the technologies that emphasize the formant and the pitch for a voice deteriorated by quantization noise according to the coding have been put to practical use until now.

Voice enhancement technologies for audio CODEC that emphasize formant and pitch have been applied to the algebraic code excited linear prediction (ACELP) method, which is an audio coding method [Reference Chen and Gersho6]. Fig. 4 is the block diagram of the post-filters used in ACELP. In short-term filter among them, the formants of voice are emphasized by applying all-pole type filter to correct the linear predictive coding forecast coefficient and expanding the peak width of the spectrum envelope in the frequency domain.

Fig. 4.

Block diagram of the post-filters used in ACELP.

In G723.1, which is the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) standard for Voice over Internet Protocol (VoIP) using ACELP, the pitch-emphasis technology is applied [7]. In ACELP, the speech generation process is modeled by the combination of the coefficient of the linear prediction filter that is equivalent to vocal tract and the excitation signal of the filter that is equivalent to the vocal chord sound. With this technology, the SNR of the pitch, which is the periodic component, is improved by applying the gain according to the delay that becomes the maximum correlation of the excitation signal. This improves the quality of the decoded voice.

D) Voice enhancement technology to surrounding noises

Today, mobile terminals are used in various places, such as station platforms and shopping centers, but it may become difficult to hear the received voice due to the surrounding noise. In the past, a mobile-terminal user had to manually adjust the volume of the reproduced sound to make the received voice easy to hear.

To solve this problem, technology that makes the received voice easy to hear by automatically emphasizing the received voice according to the surrounding noise (Fig. 5) has been developed and now used in the cellular phones and smartphones.

Fig. 5.

Outline of voice enhancement technology.

2) Features of developed voice enhancement technology

Fig. 6 shows the composition of our developed voice enhancement technology. This technology improves listenability by applying the gain to the received voice in the frequency domain according to the power of each frequency of the received voice (speech spectrum), and that of the surrounding noise obtained using the microphone in the mouthpiece of the mobile terminal (noise spectrum).

Fig. 6.

Block diagram of developed voice enhancement technology.

This technology has three features, to (i) improve the listenability of the received voice for noise of various volumes, (ii) maintain the naturalness of the voice for different qualities of speech of a caller, and (iii) make the received voice easy to hear according to the type of noise (bubble noise).

We explain these features in more detail below.

(i) Gain adaptive control based on volume ratio of surrounding noise and received voice. It is necessary to increase the volume of the enhanced voice compared with the volume and constancy of the surrounding noise to make the enhanced voice easy to hear. Then, the listenability of the received voice with respect to the surrounding noise of various volumes is achieved by analyzing the power of each frequency of the surrounding noise and received voice and controlling the gain of each frequency adaptively so that the SNRs of the surrounding noise and enhanced received voice may become higher than the prescribed value.

(ii) Gain control between bands that maintain naturalness of received voice. It was revealed that the enhanced voice is harsh and its naturalness degrades when the power of the voice at high frequencies (2–4 kHz) is much larger than that of the voice at low frequencies (0–2 kHz). Though the voice quality of the caller (power of the voice at high frequencies) varies, the naturalness of the enhanced voice can be maintained by adjusting the gain in each frequency so that the differences between the power at high frequencies and that at low frequencies are within the prescribed range.

(iii) High-frequency gain control according to the type of noise. The power of the human voice is large in a pitch frequency that originates in the vibration of the vocal chords. In the overtone frequencies that are multiples of the pitch, the frequency of vocal chords changes with time. Therefore, the pitch and overtone frequencies of the human voice change with time. Therefore, bubble noise has a feature in which the time variance of the strength of the frequency component is large. Therefore, it becomes easy to hear the received voice over a variety of noise characteristics by detecting the presence of bubble noise based on the time variance of the frequency component of the surrounding noise and amplifying the received voice at high frequencies (2.0–3.5 kHz) that is a band with aural high sensitivity when a surrounding noise is bubble noise (Fig. 7).

Fig. 7.

Principle of high-frequency gain control according to the type of noise.

3) Evaluation test

To evaluate the performance of the developed voice enhancement technology in terms of improving voice quality, we conducted a relative evaluation test concerning listenability. In this test, the received voice that was unamplified (original voice) and voice enhanced using the developed technology was compared in a noise environment. The test used the comparison category standard method, which is generally used as a speech quality measure method for VoIP [Reference Floriano, Mauro, Peppino and Salvadore8]. The voices were based on a five-point scale (−2: difficult to hear compared with the original voice, −1: slightly difficult to hear compared with the original voice, 0: listenability is equal to the original voice, +1: slightly easy to hear compared with the original voice, +2: easy to hear compared with the original voice). Forty-eight people ranging from twenties to eighties participated.

Fig. 8 shows the evaluation results. The listenability from the developed technology exceeded that (0) of the original voice for the participants under all noise conditions, and there was a significant difference in the listenability of the original voice. Therefore, the developed voice enhancement technology improved the listenability of the voice without depending on the noisy environment.

Fig. 8.

Evaluation test results (developed voice enhancement technology).

A) Background in speech rate conversion technologies

As the use of digital recorders, such as integrated circuit recorders spreads, the opportunities to record and playback voice with high sound quality has increased. The use of digital signal processor can process digital signals in real time is also spreading. As a result, the need to convert voice at different speeds and play it back with its naturalness maintained, has increased. For example, the technologies that slow the speech rate of an input voice have been developed and are used in broadcasting such as television or radio [Reference Nakamura, Seiyama, Imai, Takagi and Miyasaka9–Reference Nejime, Aritsuka, Imamura, Ifukube and Matsushima11].

Interactive speech communication in which various users talk over the telephone with each other has recently increased due to the spread of mobile terminals. Callers include those who talk fast since it is likely to talk fast in busy situations while moving etc. In such cases, the user, who is the listener, occasionally finds it difficult to hear. This is a serious problem for middle-aged people and seniors. Therefore, a speech rate conversion technology for slowing the speech rate of callers without decreasing voice quality has been developed and is used in cellular phones and smartphones.

We now give a general overview of the basic speech rate conversion technologies for mobile terminals and explain their characteristics.

B) Trends of speech rate conversion technology

Human voice includes a periodic element that originates in the vibration of the vocal chords. The pitch of the voice lowers because the cycle length changes if the speech waveform is simply enlarged in the time domain. Therefore, it is necessary to repeat the waveform with the periodic component in the voice section to maintain the voice quality of a caller (Fig. 9).

Fig. 9.

Basic principles of speech rate conversion.

As the speech rate conversion technology extends speech waveform in time domain, Synchronized OverLap Add (SOLA) method was proposed [Reference Roucos and Wilgus12]. This method searches the connecting location between frames made to overlap by not expressly detecting the periodic pattern and uses the standard based on the cross-correlation. In addition, Pitch Synchronized OverLap Add method was proposed [Reference Moulines and Charpentier13]. This method extends the voice section by detecting the pitch period from the input voice and pasting the pitch periodic components.

In these methods, by cutting out and overlapping the frame of the input speech, and adjusting the range of repetition for the connection, the speech rate can be changed with the tone of the speech kept (Fig. 10). On the other hand, the speech quality deterioration was occasionally caused when applying to the transition section where the pitch period was indistinct.

Fig. 10.

Principles of OverLap-Add method.

To solve this problem, Phase Vocoder was proposed [Reference Laroche and Dolson14]. With this technology, the periodic component is extracted by converting the voice into a frequency domain and calculating the phase of each frequency component.

C) Requirements of speech rate conversion technology to apply to mobile terminals

The following requirements occur when the basic speech rate conversion technology is applied to mobile terminals.

The first requirement is slowing the speech rate of the received voice while maintaining voice quality. The amplitude and the pitch period of natural speech waveform change with time. However, if the speech waveform is simply repeated continuously to slow the received voice, a mechanical allophone is generated because the changes at the amplitude and pitch period locally decrease, decreasing the converted voice quality. Therefore, it is necessary to slow the speech rate of the received voice while maintaining voice quality.

The second requirement is preventing the call partner from feeling uncomfortable in the conversation even if the user slows his/her speech rate. Because the reproduction time of the sent voice becomes longer than the received voice by slowing the speech rate, the time the call partner finally hears the voice is delayed. As a result, the conversation drags on and it becomes difficult to communicate in real time. Therefore, the call partner may feel uncomfortable when the gap of the conversation is too long.

D) Features of developed speech rate conversion technology

Fig. 11 shows a block diagram of the developed speech rate conversion technology. It detects the voice section and silent section (where the voice is not included) included in the received voice, and the speech rate of the received voice is slowed by shortening the silent section while extending the voice section.

Fig. 11.

Block diagram of developed speech rate conversion technology.

The developed technology has two features, (1) extending the voice without changing voice quality and (2) preventing the call partner from feeling uncomfortable in the conversation even if the user slows his/her speech rate.

2) Delay control that does not lead to obstruct conversation even if speech rate is slowed

We investigated the length of delay in an interactive telephone call when the listener felt that there was an obstacle in the conversation. We assumed there is no obstacle in a conversation if the delay is shorter than one second. The developed technology shortened the silent section to reduce delay and gradually returned the speech rate to normal when the delay was one second or longer (Fig. 12). The call partner did not experience any obstacle in the conversation because the delay according to the voice extension was within one second, resulting in real-time maintenance of the interactive telephone call.

Fig. 12.

Example using developed speech rate conversion technology.

The delay is reduced by shortening the waveform in the silent section while extending the waveform in the voice section. If the delay is over one second, the waveform in the voice section is not extended (returns to original speech rate).

E) Evaluation test

To evaluate the effectiveness of the developed speech rate conversion technology in term of improving voice quality, we conducted a relative evaluation test concerning listenability. The processing sound that slowed the speech rate using the developed technology and the received voice of original speech rate (original voice) were compared.

The evaluation was measured based on a seven-point scale (−3: very difficult to hear compared with an original voice, −2: difficult to hear compared with the original voice, −1: slightly difficult to hear compared with the original voice, 0: listenability equal to the original voice, +1: slightly easy to hear compared with the original voice, +2: easy to hear compared with the original voice, and +3: very easy to hear compared with the original voice). Twenty-eight people ranging in age from thirties to eighties participated. The speech rate of the original voice was 150–200 words per minute, which is considered fast for middle-aged people and seniors.

Fig. 13 shows the evaluation results. The listenability with the developed technology exceeded listenability (0) of an original voice for the participants in all age groups, and there was a significant difference in the listenability of the original voice. Therefore, the developed speech rate conversion technology improved listenability no matter the age of the listener when the speech rate of the received voice was fast.

Fig. 13.

Evaluation test results (developed speech rate conversion technology).

A) Attaining high performance of developed technologies at various speech qualities

The received voice quality depends on the speech quality of the call partner, acoustic characteristics of mobile terminal, and transmission path (Fig. 14). Without speech control, our developed technologies will not perform successfully and speech quality will vary according to the conditions. (Fig. 15a). We have developed a received-voice-quality-control technology. With this technology, we can adjust the received voice quality and achieve the desired performance of our voice enhancement and the speech rate conversion technologies (Fig. 15b).

Fig. 14.

Factors of received speech quality.

Fig. 15.

Attaining “standard speech quality” with adaptive speech quality control. (a) Without adaptive speech control (upper). (b) With adaptive speech quality control (lower).

Fig. 16 shows a diagram of this technology. It first analyzes the received voice frequency characteristics then calculates corrected gains to match reference values. By applying these corrected gains to the received voice, preferable speech quality will be attained. For example, when the power of lower frequencies is larger than that of higher frequencies, muffled speech will result. We define preferable speech quality for mobile terminals as “standard speech quality”. It is a “rich, deep resonant tone”. With this technology, we adjust the frequency component to implement “standard speech quality” for cellular phones.

Fig. 16.

Diagram of speech quality control.

B) Attaining high performance of developed technologies in noisy environment

Cellar phones have to provide good performance under various environments, from noisy streets and restaurants to quiet rooms. In noisy environments, speech cannot be heard over the noise (SNR degradation). In this condition, speech quality becomes increasingly difficult to maintain. Our speech rate conversion technology slows the voice in order to easier listening. To recover from the delay, this technology detects and deletes silence between speeches. Therefore, voice activity detection (VAD) affects the performance of the technology.

VAD detects voice that has different characteristics than environmental noise. It uses SNR as one of the characteristics. It adaptively controls the threshold to detect voice depending on the strength of the environmental noise and calculates the sound characteristics for VAD using high SNR frequency components. We expect that high SNR frequency components will be less affected by environmental noise than low SNR frequency components. This will enable VAD to provide high performance under noisy environments (Fig. 17). VAD detects more than 60% silence duration under noisy environments, improving the effectiveness of our speech rate conversion technology.

Fig. 17.

VAD mechanism.

A) Voice enhancement technology based on user movement

When we are moving, commuting, going to school, and shopping, it is likely we talk on our mobile terminals while walking or running. In such a situation, it might become difficult to hear the received voice because the ear piece of the mobile terminal tends to leave from the user's lughole by the user's head and arm moving along with walking. Therefore, a technology for amplifying the strength of each frequency of the received voice according to the user's movement (walking/running) has been developed. This technology detects the degree of shake (movement) using the motion sensor installed in mobile terminals and determines whether the user is walking or running. Listenability improves by amplifying the strength of each frequency of the received voice according to the determined movement.

B) Voice enhancement technology based on user age

It is generally known that hearing becomes difficult as a person ages and the ability to hear voices decreases (Fig. 18). Therefore, a technology that improves listenability by amplifying the received voice according to the user's hearing ability has been developed.

Fig. 18.

Hearing ability decreases with increasing age.

The features of development technology will be described below.

2) Amplification amount control according to input volume

For seniors, it is difficult to hear faint sounds, but loud noises can be heard. Therefore, simply amplifying all sounds would be annoying to seniors. Therefore, when the input volume (volume of the received voice) is low, the input voice is amplified based on the hearing loss level, as shown in Fig. 19. When the input volume is high, the input voice is amplified to not exceed the volume at which unpleasantness is experienced. Through this control, such unpleasantness is eliminated.

Fig. 19.

Amplification amount control according to input volume. Sound pressure level (SPL).