← Back

Jul 20, 2026 · 13 min read

Clippings #3

This edition of Clippings covers links that I collected between May and July 2026. I made the decision to discard the fixed monthly cadence for Clippings. I hadn’t collected enough interesting links in May, so I extended it to June. But then I didn’t have time to generate the report until late July. It felt more natural to move Clippings to a numbered issue and drop the monthly requirement altogether. I retroactively renamed the previous editions to #1 and #2.

In this edition, I’m featuring three interesting releases from Kyutai: post-training for full-duplex models, MoshiRAG, and PocketTTS. I’m also including a survey on Spoken Language Models, and Lilian Weng’s blog post on Scaling Laws. I found it interesting how the survey connects to the blog post by noting that we still generally lack a good understanding of scaling laws for speech.

Highlights

Post-training speech models for better interactivity
Full-Duplex RL Post-Training Interactivity Alignment

The paper proposes an RL post-training stage to improve the interactive behavior of full-duplex speech models after supervised fine-tuning. Instead of learning a single conversational reward, it decomposes interactivity into four behaviors—pause handling, turn-taking, backchanneling, and interruption handling—and constructs a separate training set and reward function for each. Most rewards are rule-based and rely on voice activity detection (VAD) to evaluate when the model speaks rather than what it says. Specifically, pause handling penalizes unnecessary speech during brief user pauses; turn-taking and interruption handling reward fast responses by penalizing response latency; and backchanneling uses an F1-based reward that measures whether short acknowledgments occur near human backchannel timings. To preserve response quality, the authors supplement these timing-based rewards with an LLM-based semantic reward, which evaluates the appropriateness of the generated content. Together, these rewards are optimized using GRPO, aligning the model’s conversational timing and interaction patterns without sacrificing semantic quality.

The paper has a companion blog post and the post-trained weights are released as an HF collection .

A full-duplex speech model can simultaneously listen to the user and generate speech, enabling real-time, bidirectional conversations without requiring strict turn boundaries. Unlike traditional half-duplex systems that alternate between listening and speaking, full-duplex models can naturally handle overlapping speech, interruptions, backchanneling, and fluid turn-taking, making interactions more human-like.

MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models
Full-Duplex RAG

The paper proposes MoshiRAG, a retrieval-augmented generation (RAG) framework specifically designed for full-duplex speech systems. MoshiRAG extends compact full-duplex speech models with RAG while preserving a natural conversational behavior. Rather than delaying a response until retrieval completes, the model learns to emit a special retrieval token (<ret>) that asynchronously triggers an external retrieval backend. The speech model immediately begins speaking (e.g., “That’s a great question…”), effectively hiding retrieval latency. Once the retrieved information becomes available, it is encoded into a compact latent representation using a reference text encoder and injected into the model’s hidden states, allowing the model to seamlessly transition into a grounded response without restarting generation. This generate–retrieve–inject paradigm differs from conventional retrieve-then-generate RAG pipelines by overlapping retrieval and response generation. A useful design decision is that the RAG pipeline is independent of the main speech model, and it can be swapped without retraining the main model.

Code is available on GitHub , alongside Kyutai’s blog post walking through the design.

Pocket TTS: a high-quality TTS with voice cloning that runs on CPU
TTS On-Device Continuous Audio LM

Pocket TTS is a lightweight, 100M-parameter streaming text-to-speech model. It is open-source and can run locally on standard CPUs. Its key feature is the use of Continuous Audio Language Models , which generate speech from continuous audio representations rather than from discrete codec tokens. Using continuous representations helps reduce the impact of autoregressive decoding and improves inference time. The model is combined with a speaker encoder that extracts voice characteristics from just a few seconds of reference audio. The blog post claims that Pocket TTS can synthesize natural speech in unseen voices without fine-tuning. The post also hints at an updated version of the “Continuous Audio Language Models” paper, with more details on Pocket TTS.

The code and models can be found on GitHub .

On The Landscape of Spoken Language Models: A Comprehensive Survey
Survey Spoken Language Models

The paper surveys the emerging field of Spoken Language Models (SLMs) and proposes a taxonomy that organizes existing approaches into three categories: Pure Speech Language Models, which model speech directly using discrete speech tokens and preserve acoustic information such as prosody, emotion, and speaker identity; Speech-Aware Text Language Models, which extend pretrained text LLMs with speech encoders to leverage their reasoning capabilities while often relying on implicit speech-to-text representations; and Joint Speech–Text Language Models, which learn unified representations of speech and text to enable more natural multimodal interaction.

Beyond this taxonomy, the survey highlights several key challenges facing the field:

  • the lack of a standard definition of SLMs
  • unresolved questions around speech tokenization and representation
  • fragmented evaluation benchmarks
  • limited understanding of scaling laws
  • scarcity of large, high-quality multilingual speech datasets
  • the difficulty of achieving low-latency, human-like conversational interaction
  • growing concerns around safety, privacy, and voice impersonation

Overall, the paper argues that the field is transitioning from cascaded pipelines toward speech-native foundation models, while emphasizing that significant research is still needed before spoken language models reach the maturity of modern text LLMs.

Spoken language models (SLMs) are foundation models that directly model spoken language by learning from speech representations, enabling them to understand and/or generate speech without relying exclusively on intermediate text. Unlike traditional speech processing pipelines that separate speech recognition, language understanding, and speech synthesis, SLMs aim to unify these capabilities while preserving the rich acoustic information in speech, including prosody, speaker characteristics, emotion, and timing.

Scaling Laws, Carefully
Scaling Laws Survey Deep Dive

In this blog post, Lilian Weng revisits the foundational results of Scaling Laws and surveys subsequent advances. The post argues that scaling laws remain indispensable for forecasting model performance and guiding resource allocation, but should be understood as empirical models rather than universal laws. The author emphasizes that the predictive power of scaling laws depends on the assumptions under which they are derived and that emerging challenges—including finite high-quality data, repeated data usage, new model architectures, and inference-time compute—require extending and refining classical scaling laws for modern AI systems.

Scaling laws are empirical relationships that describe how the performance of machine learning models changes as resources such as model size, training data, and compute are increased. Two key references for scaling laws are Kaplan et al. (2020): Scaling Laws for Neural Language Models and Hoffmann et al. (2022): Training Compute-Optimal Large Language Models (often called the “Chinchilla” paper). The work of Kaplan et al. (2020) demonstrated that language model performance follows predictable power-law trends, while Hoffmann et al (2022), “Chinchilla”, showed that compute-optimal training often favors smaller models trained on substantially more data rather than simply increasing parameter counts.

Research Papers

Speech & Audio

  • Stable Audio 3 A family of fast latent diffusion models (small, medium, large) for variable-length audio generation and editing. Prior latent diffusion audio generators typically had to produce full-length outputs even for short sounds; Stable Audio 3’s variable-length generation avoids paying that fixed cost regardless of the target duration. The result is a single model family that scales from short sound effects to multi-minute generations without a separate long-form pipeline.

  • Position: Towards Responsible Evaluation for Text-to-Speech TTS systems now often generate speech indistinguishable from human recordings, but the paper argues evaluation practices haven’t kept pace with that shift. It’s a position paper making the case for evaluation protocols that account for the accessibility, content authenticity, and human-computer-interaction risks that near-perfect synthetic speech introduces, rather than continuing to rely on naturalness/MOS-style metrics alone.

  • Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM Speech LLMs are usually built by adapting a text LLM checkpoint, but they retain a persistent modality gap relative to the text model. Most prior fixes attack the gap from the output side, making speech generation more text-like; this paper instead reduces the gap from the input side, by making the text LLM itself prosody-aware before speech is introduced. Framing the fix as an input-side change rather than an output-side one is the paper’s central departure from prior work.

  • Mega-ASR: Towards In-the-wild² Speech Recognition via Scaling up Real-world Acoustic Simulation Real-world ASR and audio-LLMs suffer from an “acoustic robustness bottleneck” — under severe, compositional noise conditions, models lose acoustic grounding and start hallucinating or omitting content rather than degrading gracefully. Mega-ASR attacks this by scaling up real-world acoustic simulation used in training, rather than relying on cleaner or more limited noise augmentation. The paper frames this as necessary for models to remain trustworthy outside curated benchmark conditions.

  • LLM can Read Spectrogram: Encoder-free Speech-Language Modeling Speech-aware LLMs almost universally rely on a pre-trained speech encoder to convert raw audio into representations the language model can consume. This paper asks whether that encoder is necessary at all, training an LLM to read Mel spectrograms directly the way it reads text tokens. Removing the encoder collapses the usual two-stage pipeline (encoder pretraining, then LLM adaptation) into a single model, simplifying the architecture and eliminating a component the rest of the system otherwise has to be adapted around.

  • Segmental Attention Decoding With Long Form Acoustic Encodings Attention-based encoder-decoder (AED) ASR models are typically trained on segmented utterances and learn to implicitly encode absolute frame positions by exploiting the limited acoustic context available within a segment — a shortcut that breaks down on long-form audio the model wasn’t trained on. The paper’s segmental attention decoding addresses this incompatibility directly, aiming to let AED models generalize to long-form acoustic encodings without the positional shortcut failing.

LLM Architecture

  • Locally Coherent Parallel Decoding in Diffusion Language Models | Diffusion language models offer sub-linear generation latency and bidirectional context versus autoregressive models, but achieving that sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel — which tends to break local coherence between neighboring tokens. This paper targets that specific failure mode, aiming to keep parallel decoding locally coherent rather than trading coherence for speed.

  • Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — A single model architecture that can operate in autoregressive, diffusion, or self-speculative decoding modes, rather than committing to one decoding paradigm at training time. Self-speculation lets the model draft and verify its own tokens without a separate draft model, unifying three normally distinct inference strategies under one set of weights.

  • Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini — A native multimodal embedding model built on Gemini that embeds video, audio, image, and text — including arbitrary interleaved combinations of those modalities — into a single unified representation space, rather than bolting separate per-modality encoders together.

  • MeMo: Memory as a Model — LLMs are frozen after pretraining and need efficient mechanisms to incorporate new, timely, domain-specific information without a full retrain. MeMo proposes treating memory itself as a model component that can be updated, rather than relying purely on retrieval over an external store or repeated fine-tuning.

Training & Compression

  • Self-Distillation Enables Continual Learning — On-policy reinforcement learning can reduce catastrophic forgetting during continual learning, but it requires explicit reward functions that are often unavailable for new skills. The paper shows self-distillation can substitute for that reward signal, enabling continual learning without hand-designed rewards.

  • Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs | [Weights ] — A compressed variant of Nemotron-3-Super, optimized specifically for interactive serving throughput on a single 8×B200 node rather than raw parameter efficiency alone.

  • TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate | [Code ] — An online vector quantization method addressing both mean-squared-error and inner-product distortion simultaneously, rather than optimizing for one at the expense of the other; turbovec is a Rust vector index (with Python bindings) built on top of it.

Analysis, Safety & Agents

  • Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew — Current training data attribution (TDA) methods for generative models lack the reliability needed for real-world adoption. The paper proposes a mirrored-unlearning approach with a noise-consistent skew correction to make TDA in diffusion models more robust.

  • AutoHarness: improving LLM agents by automatically synthesizing a code harness — LLM agents frequently attempt actions that are strictly prohibited by their environment rather than merely suboptimal. AutoHarness automatically synthesizes a code harness around the agent to constrain it to valid actions, rather than relying on the policy alone to learn environment constraints.

  • From AGI to ASI — A position paper from a DeepMind-affiliated author group examining the path from human-level AGI to artificial superintelligence, and the societal impacts that transition raises.

  • StoryScope: Investigating idiosyncrasies in AI fiction — As AI-generated fiction becomes more prevalent, questions of authorship and originality are becoming central to how written work is evaluated. Rather than focusing on surface-level AI-writing signatures, StoryScope asks whether AI-generated stories exhibit deeper structural idiosyncrasies.

Also Worth a Look

Clippings #2 →