How to build high-performance speech synthesis systems with compact speech representations

The Xiaohongshu Multimedia Intelligent Algorithm Team and the Chinese University of Hong Kong jointly proposed a high-performance speech synthesis scheme MSMC-TTS based on multi-stage multi-codebook compact speech representation for the first time. The feature analyzer based on vector quantized variational autoencoder (VQ-VAE) uses several codebooks to encode acoustic features in stages to form a set of latent sequences with different temporal resolutions. These latent sequences can be predicted from text by a multi-stage predictor and converted into target audio by a neural vocoder. Compared with the Mel-Spectrogram-based Fastspeech baseline system, this solution has significant improvements in sound quality and naturalness. This work has now been summarized into the paper "A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS" and was accepted by the speech domain conference INTERSPEECH 2022.

1. Background introduction

Text-to-Speech (TTS) is a technology that converts text into speech. It is widely used in video dubbing and audio and video content creation. , intelligent human-computer interaction and other products. The back-end acoustic modeling technology of mainstream speech synthesis systems usually includes three parts: feature extractor, acoustic model and vocoder. TTS usually performs acoustic modeling on acoustic features obtained based on signal processing (such as Mel Spectrogram). However, limited by the fitting ability of the model, there is a certain difference in distribution between the predicted acoustic features and the real data. This This makes it difficult for vocoder trained on real data to generate high-quality audio from predicted features.

##TTS system framework diagram

To address this problem, the academic community uses A more complex model structure and a more novel generative algorithm are adopted to reduce prediction errors and distribution differences. This work takes a different approach, taking compact speech representation as the starting point to consider the problem. For speech synthesis, 1) good compactness of acoustic features can ensure more accurate model prediction results and more robust waveform generation; 2) good completeness of acoustic features can ensure better reconstruction of speech signals. Based on these two considerations, this paper proposes to use vector quantization variational autoencoder (VQ-VAE) to mine a better compact representation from the target data.

2. Representation learning MSMC VQ-VAE

VQ-VAE includes encoder and decoder. The encoder processes the input acoustic feature sequence into a latent sequence and quantizes it using the corresponding codebook. The decoder restores the quantized sequence to the original acoustic feature sequence. This quantized sequence has better compactness (less number of characteristic parameters) as a discretized representation. The higher the degree of quantization, that is, the smaller the codebook capacity, the higher the compactness of the features. But this also results in information compression, making feature completeness worse. In order to ensure sufficient completeness, more codewords are generally used. However, as the codebook capacity increases, the amount of data required for codebook update and the number of training times will increase exponentially, which makes it difficult for VQ-VAE to effectively enhance the representation completeness by increasing the codebook. To address this problem, this paper proposes the multi-head vector quantization (MHVQ) method.

##VQ-VAE model structure diagram

MHVQ will be a single codebook It is divided into several sub-codebooks according to the direction of feature dimensions. During quantization, each input vector is equally cut into several sub-vectors, quantized with corresponding sub-codebooks respectively, and finally spliced ​​into an output vector. In this way, we can more effectively improve codebook utilization and representation capacity without increasing the amount of codebook parameters. For example, to reduce the compression ratio by a factor of 1, the codewords must be increased to the square of the original codebook number. After using MHVQ, the same compression rate can be achieved by simply splitting the codebook into two parts. Therefore, this method can regulate the completeness of quantitative representation more effectively.

MHVQ example picture

In addition, when quantizing the speech sequence , all kinds of information contained in speech features are lost to varying degrees. This information is different in time granularity, such as coarse-grained timbre, pronunciation style, etc., and fine-grained pitch, pronunciation details, etc. Over-compressing information on any time scale can cause some degree of degradation in speech quality. To alleviate this problem, this work proposes a multi-time scale speech modeling method. As shown in the figure, the acoustic feature sequence is encoded in stages to different time scales through several encoders, and then quantized layer by layer through the decoder, decoding to obtain several quantized sequences with different time resolutions. The representation composed of this type of sequence set is the multi-stage multi-codebook representation proposed in this work.

Multi-stage modeling example diagram

3. Acoustic modeling MSMC-TTS

For multi-stage Multi-codebook represents MSMCR, and this paper proposes the corresponding TTS system, namely MSMC-TTS system. The system includes three parts: analysis, synthesis and prediction. In system training, the system first trains the analysis module. The audio in the training set is converted into high-completeness acoustic features (such as the Mel-Spectrogram features used in this work) after signal processing. These acoustic features are used to train the feature analyzer based on MSMC-VQ-VAE. At the end of the training, they are converted into the corresponding MSMCR, and then the acoustic model and neural vocoder are trained. During decoding, the system uses an acoustic model to predict MSMCR from text and then uses a neural vocoder to produce the target audio.

##MSMC-TTS system framework diagram

This work also proposes a A multi-stage predictor to fit MSMCR modeling. This model is implemented based on FastSpeech, but differs on the decoder side. The model first encodes the text and upsamples the text based on predicted duration information. The sequence is then downsampled to each time resolution corresponding to MSMCR. These sequences will be decoded and quantized step by step from low resolution to high resolution by different decoders. At the same time, the low-resolution quantized sequence is sent to the next stage decoder to assist prediction. Finally, the predicted MSMCR is fed into the neural vocoder to generate the target audio.

Multi-stage predictor structure diagram

For multi-stage predictor When performing training and inference, this work chooses to directly predict the target representation in continuous space. This method can better take into account the distance relationship between vectors and codewords in linear continuous space. In addition to the MSE loss function commonly used for TTS modeling, the training criterion also uses a "triplet loss" to force the prediction vector away from non-target codewords and closer to the target codeword. By combining the two loss function terms, the model is able to better predict the target codeword.

4. Experimental results

This work was conducted on the public English single-speaker data set Nancy (Blizzard Challenge 2011). We organized a subjective opinion score test (MOS) to evaluate the MSMC-TTS synthesis effect. The experimental results show that when the original recording is 4.50 points, the MSMC-TTS score is 4.41 points, and the baseline system Mel-FS (Mel-Spectrogram based FastSpeech) is 3.62 points. We tuned the vocoder of the baseline system to match the Mel-FS output characteristics, and the result was 3.69 points. This comparison result proves the significant improvement of the TTS system proposed by the method proposed in this article.

In addition, we further discussed the impact of modeling complexity on the performance of TTS. The number of model parameters decreases exponentially from M1 to M3, and the Mel-FS synthesis effect drops to 1.86 points. In contrast, for MSMC-TTS, the reduction in the number of parameters did not have a significant impact on the synthesis quality. When the acoustic model parameter size is 3.12 MB, the MOS can still maintain 4.47 points. This not only demonstrates the low complexity requirements of MSMC-TTS modeling based on compact features, but also demonstrates the potential of this method to be applied to lightweight TTS systems.

Finally, we conducted a MSMC-TTS comparison based on different MSMCRs to explore the impact of MHVQ and multi-stage modeling on TTS. . Among them, the V1 system uses a single-stage single codebook representation, the V2 system uses 4-head vector quantization based on V1, and the V3 system uses two-stage modeling based on V2. First, the representation used by the V1 system has the highest feature compression ratio, but exhibits the lowest completeness in analysis synthesis experiments and the worst synthesis quality in TTS experiments. After MHVQ enhanced the completeness, the V2 system has also been significantly improved in TTS effect. Although the multi-stage representation used by V3 did not show further improvements in completeness, it showed the best results on TTS, with significant improvements in both rhythmic naturalness and audio quality. This further shows that multi-stage modeling and multi-scale information retention are of great significance in MSMC-TTS.

5. Summary

This work proposes a new set of high-performance TTS (MSMC-TTS) modeling methods from the perspective of studying compact speech representation. The system extracts multi-stage multi-codebook representations from audio in place of traditional acoustic features. Input text can be converted into this speech representation consisting of multiple sequences with different temporal resolutions by a multi-stage predictor and converted to a target speech signal by a neural vocoder. Experimental results show that compared with the mainstream FastSpeech system based on Mel-Spectrogram, this system exhibits better synthesis quality and lower requirements for modeling complexity.

6. Author information

Guo Haohan: Intern in Xiaohongshu Multimedia Intelligent Algorithm Team. He graduated from Northwestern Polytechnical University with a bachelor's degree and studied in the ASLP laboratory under Professor Xie Lei. Currently, he is studying for his Ph.D. in the HCCL Laboratory of the Chinese University of Hong Kong, studying under Professor Meng Meiling. So far, as a first author, six papers have been published at ICASSP, INTERSPEECH, and SLT international speech conferences.

Xie Fenglong: Head of voice technology of Xiaohongshu Multimedia Intelligent Algorithm Team. He has published more than ten papers in speech conferences and journals such as ICASSP, INTERSPEECH, and SPEECHCOM. He has long been a reviewer for major speech conferences such as ICASSP and INTERSPEECH. His main research direction is speech signal processing and modeling.

The above is the detailed content of How to build high-performance speech synthesis systems with compact speech representations. For more information, please follow other related articles on the PHP Chinese website!