Clippings - April 2026

OmniVoice is a zero-shot TTS model covering more than 600 languages, and it is trained on 581k hours of open-source multilingual audio. The system uses a single-stage non-autoregressive diffusion-based model architecture. Specifically, OmniVoice is built on Qwen3-0.6B as the bidirectional transformer backbone (~0.8B total params) and on the Higgs-audio tokenizer (8-codebook acoustic tokens). Generation is done with a 32-step iterative unmasking loop.

There are two key contributions in this paper. The first is full-codebook random masking, where every token in the TxC matrix is independently masked, with 50% of the tokens used in the loss function. The second is the initialization of the LLM, which is based on pre-trained AR LLM weights (Qwen3-0.6B). According to the authors, initialization from an AR model helps improve intelligibility (measured through word error rate).

Weights are released under Apache 2.0, with code on GitHub (k2-fsa/OmniVoice ) and a live demo on HuggingFace Spaces .

Non-autoregressive (NAR) models generate all output tokens in parallel rather than one at a time like standard autoregressive (AR) models. The trade-off is between sequential dependencies and speed. In TTS, NAR architectures typically suffer from quality degradation compared to autoregressive counterparts, where AR architectures can be slow, but more stable.

Alibaba’s Qwen team releases Qwen3-TTS, an open-source multilingual TTS series with 3-second voice cloning, description-based voice control, and streaming generation. Qwen3-TTS includes 0.6B and 1.7B models, with the series varying supported features.

The models are released with two specialised tokenizers: 1) Qwen-TTS-Tokenizer-25Hz, leveraging Qwen-Audio, this is a single-codebook representation focused on semantics with reconstruction via a diffusion transformer with block-wise flow-matching; and 2) Qwen-TTS-Tokenizer-12Hz, multi-codebook representation focusing on extreme bit-rate reduction, with reconstruction via a causal ConvNet decoder.

Qwen3-TTS is based on the Qwen3 LM family, interleaving text and audio tokens. Training follows a standard pipeline consisting of pre- and post-training, leveraging techniques such as Direct Preference Optimization (DPO) and Group Sequence Policy Optimization (GSPO).

The model series is released with code on GitHub .

DPO is a fine-tuning method that trains a model directly on preferred vs. rejected response pairs, bypassing the need for a separate reward model. See the DPO paper.

GSPO is a PPO-style reinforcement learning algorithm that computes importance ratios at the sequence level (length-normalized) rather than per-token, making training more stable by weighting all tokens in a response equally. See the GSPO paper.

MOSS-TTS is an open-source speech generation foundation model. The technical report introduces the MOSS-Audio-Tokenizer and the MOSS-TTS system. The tokenizer is a 1.6B causal Transformer that compresses 24 kHz audio to 12.5 frames per second using a multicodebook representation with 32 RVQ layers. The model is trained on millions of hours of diverse audio, including speech, music, and noise.

The model backbone is a standard auto-regressive transformer language model with interleaved text and audio tokens. There are two approaches to modelling: 1) a delayed pattern strategy to RVQ tokenization, simpler to implement, but requiring N timesteps for N RVQ codebooks; and 2) a depth transformer to generate full RVQ codebooks from a global embedding at each time-step.

The report provides extensive details on data preparation for large-scale pre-training (preprocessing, filtering, synthesis) and comprehensive evaluations for the proposed audio tokenization method and TTS across various tasks.

Code and weights are publicly available .

In multi-codebook codec models, each audio frame is represented by K tokens — one per RVQ level. The delayed pattern extends these across time in a staircase pattern. See Figure 1 in the MusicGen paper. This turns an otherwise parallel K-token prediction into a standard left-to-right autoregressive stream without any architectural special-casing. The depth transformer, on the other hand, uses a small secondary transformer within each frame, modelling the conditional dependencies between codebook levels at that single timestep. See Figure 3 in the Moshi paper. Together they offer two complementary approaches: delay flattens the codebook dimension into the time axis; depth keeps it separate but handles it with a dedicated, lightweight model.

VoXtream2 is a zero-shot full-stream model where the rate can be updated mid-utterance on the fly, while the system continues generating audio with minimal delay as text arrives incrementally.

The VoXtream2 model is an autoregressive transformer that is conditioned on sequences of phonemes (given by a lexicon or a grapheme-to-phoneme component), and generates duration tokens enforcing a monotonic alignment between text and speech. The model consists of a primary temporal transformer, and a depth transformer to handle multi-codebook prediction. The codebooks are based on the Mimi tokenizer, introduced in the Moshi paper. The VoXtream2 model further uses classifier-free guidance (CFG) and masks text and audio conditioning signals during training with a probability of 10%. To adjust speaking rate dynamically during inference, the system manipulates the probability distributions of the duration states.

Model weights, code, a live demo , and a project page are all openly available.

“A full-stream TTS system is an architecture that operates entirely in streaming mode, accepting incrementally generated text tokens as input and producing small waveform chunks as output. A full-stream system begins generating speech before the full text is available, minimizing first packet latency.” (Section 2.2 of the paper).

VibeVoice, an ICLR 2026 Oral from Microsoft, targets long-form multi-speaker podcast generation. This scenario can be challenging for speaker consistency across extended segments.

The paper describes two speech “tokenizers”, operating in continuous space. The Acoustic Tokenizer is based on Variational AutoEncoders (VAEs), specifically the σ-VAE variant. This tokenizer focuses on speech generation. The Semantic tokenizer is simpler and fully-deterministic, and focuses on speech understanding.

The VibeVoice model architecture uses an auto-regressive large language model with a diffusion head. It generates continuous acoustic representations at each timestep. The audio is then encoded with the semantic “tokenizer”, and both acoustic and semantic representations are combined through learnable projections. This is what the authors call “hybrid speech representations”, a mixture of both acoustic and semantic representations. Note that VibeVoice operates in a chunking-like format, with each timestep generating a chunk of audio that is concatenated to form the final rendered speech.

The system is open-source on GitHub , with the full technical report available on OpenReview .

Next-token diffusion combines the sequential generation of autoregressive language models with the continuous sampling of diffusion models. Language models predict one discrete token at a time from a fixed vocabulary. Next-token diffusion applies this to continuous latent spaces. Instead of predicting a discrete token via softmax, the model’s hidden state at each step is used to condition a lightweight diffusion head that iteratively generates a continuous latent vector. It is this continuous latent vector that is used to condition the language model in the next timestep.

Affectron tackles the generation of nonverbal vocalizations like laughter, sighs, or hesitations, as part of emotional speech synthesis. Current systems rarely model NVs directly because labelled data is scarce and there is no obvious supervision signal for contextual alignment.

Affectron uses EnCodec, a multi-codebook codec, and VoiceCraft, a standard autoregressive language model generating codebooks with a delayed pattern. The main contribution is a data augmentation mechanism to inject nonverbal vocalizations onto a verbal dataset. The backbone language model is then fine-tuned on this augmented dataset.

The approach requires a dataset of nonverbal vocalizations and a dataset of verbal speech recordings annotated with overall emotion. The first step, Top-K NV Matching, addresses how to identify the top K candidates from the nonverbal dataset for each verbal utterance. The second step, Top-K Routing, decides where in the utterance to insert the selected NVs (e.g. before a sentence, mid-breath, at a clause boundary).

The investigation is based on the EARS corpus, which has limited speakers, and an overlap of speakers between the verbal and nonverbal datasets. Although it does show the feasibility of the mechanism, it is limited in that it doesn’t generalise easily to arbitrary verbal corpora where we don’t have matching NV recordings from the same speakers. This is acknowledged in Section 8 of the paper.

Code and a demo page are publicly available.

Nonverbal vocalizations (NVs) are vocal sounds produced by a speaker that carry affective or communicative meaning but contain no linguistic content — they are not words or phonemes. Examples include laughter, sighs, cries, gasps, filler sounds (like “uh”, “um”), chuckles, giggles, and snickers. They sit alongside speech to convey emotional state, hesitation, or social cues that words alone don’t capture.

Standard semantic speech tokenizers can be sensitive to mild acoustic perturbations, leading to variable token sequences. StableToken is a speech tokenizer designed to achieve token consistency across variable signal-to-noise ratio. The model is built on top of Whisper-large-v3, which encodes a speech waveform into a sequence of hidden states. The whisper encoder is followed by average pooling to downsample the signal, and the Voting-LFQ module. Speech is reconstructed with the whisper decoder. StableToken is based on two dependent contributions: 1) a Voting-LFQ module (voting look-up-free quantizer), and 2) a noise-aware consensus training strategy.

The Voting-LFQ module runs multiple projection branches in parallel, with each branch providing a binary representation over the available tokens. Averaging over all branches gives a more reliable representation of the token sequence for the input sequence. At training time, the real-valued average is used directly by the model, whereas at inference time, the system performs a bit-wise majority vote.

This module is trained with a noise-aware consensus strategy. During each forward pass, a randomly selected minority of branches receives a noise-perturbed version of the hidden state, while the majority receive the clean version. A consensus loss then penalizes each branch’s deviation from the global average, thus ensuring that the token sequence is consistent even in the presence of noise.

Code is available on GitHub .

For speech tokenization, Vector Quantization (VQ) maps a continuous representation to a nearest entry in a finite learned codebook, producing a single discrete token. Residual Vector Quantization (RVQ) extends this by stacking multiple VQ layers in sequence. While the first layer quantizes the input, the second quantizes the residual error left by the first, and so on. Look-Up Free Quantization (LFQ) is a different approach where each input is binarized dimension-by-dimension with a sign function. The binary code directly relates to an implicit codebook of size 2^d, where d is the hidden dimension size.

Chain-of-Details TTS is a non-autoregressive TTS system that reframes the standard coarse-to-fine generation paradigm along the temporal axis rather than the codebook depth axis.

Related work progressively refines RVQ quantization layers, which means such models iteratively decode along the codebook depth axis. CoD-TTS instead generates speech at increasing temporal resolutions, starting from a low token-rate sequence, then doubling the rate across successive levels to fill in finer acoustic detail. A single shared transformer decoder based on Llama with bidirectional attention handles all resolution levels. The system is conditioned at each stage on fixed signals like G2P phoneme sequences, speaker embedding, and a duration sequence. Within each level, generation is parallel: the decoder runs masked token modeling over a fixed number of steps, iteratively unmasking the most confident positions. The paper doesn’t seem to be claiming SOTA results, but rather competitive results on a 263M base model when compared to a 476M parameter two-stage baseline.

In masked token modeling, rather than generating tokens one at a time left-to-right, the model starts with a fully masked sequence and iteratively reveals tokens over a small fixed number of steps. At each step, the model predicts all masked positions simultaneously using bidirectional attention, where every position can attend to every other. The model then commits the most confident predictions and iterates on the remaining masked positions. A cosine schedule controls how many tokens are unmasked at each step. The number of decoding steps is fixed and tends to be small, so generation is much faster than autoregressive decoding. The main drawback is that the sequence length needs to be known upfront, so systems usually rely on separate duration models for speech generation.

T5Gemma-TTS is a 4B parameter encoder-decoder text-to-speech model. The model is built on top of a 2B parameter T5 encoder and 2B parameter Gemma decoder. The model uses subword inputs rather than phoneme sequences, and it relies on the XCodec2 single-codebook tokenizer. The model applies Progress-Monitoring Rotary Position Embedding (PM-RoPE), proposed in the VoiceStar paper. The model is trained on English, Japanese, and Mandarin Chinese, and also evaluated on Korean, French, and German. This appears to be a straightforward model report with limited novelty. Perhaps the most interesting addition is the return to an encoder-decoder structure, and the validation of the PM-RoPE mechanism in a multilingual scenario.

Code and model weights are available on GitHub .

Rotary Position Embedding (RoPE) encodes positional information into attention queries and keys by rotating their vector representations by an angle proportional to a position index, such that the dot-product attention score between any query-key pair naturally reflects their relative distance. Progress-Monitoring RoPE (PM-RoPE) extends this by replacing raw token indices with normalized progress fractions, mapping each token’s position to a value proportional to how far through its respective sequence it sits. Cross-attention scores between a decoder query and an encoder key reflect alignment in progress rather than absolute position. This allows an autoregressive decoder to continuously track how far through the target output it has progressed relative to the input, enabling explicit length and duration control without requiring any supervised duration labels during training.