在語言領域的連續擴散模型（continuous diffusion models）近期出現了大量的活動，在經歷了幾年的相對沉寂後，這種方法似乎正在捲土重來。完全離散的擴散方法在很大程度上已經取代了早期試圖讓連續擴散模型適用於語言的嘗試，但潮流正在轉變。在這篇文章中，我想仔細探討正在發生的事情，以及為什麼現在會發生。
這個領域新研究的湧入激發了我寫下一些想法。我以前寫過擴散語言模型，所以這主要是作為一個更新，涵蓋了從那時起發生的一切。這將是一個相當主觀的敘述——其他觀點和不同意見非常歡迎在評論區及其他地方提出！我稍後將討論連續擴散模型在語言方面的技術細節，但首先，是一些歷史背景。
現代語言模型，總體而言，是自迴歸的（autoregressive）：它們一次生成一個 token 的序列。這是將一個困難的生成任務自然分解為更小、更容易的順序步驟。所有步驟都是相同底層任務的實例（在給定前導 token 的情況下預測一個 token），這使得參數可以在序列維度上共享。儘管有這種本質上是順序的生成過程，但 Transformer 架構 1 能夠透過 teacher forcing 2 在所有序列位置上進行高效的並行訓練。這被證明是一種極具擴展性的方法 3，為我們帶來了大型語言模型（LLM）。
然而，自迴歸並非構建序列迭代生成過程的唯一方法。受到視聽領域早期成功的啟發，研究人員試圖將擴散模型應用於語言生成。擴散模型的生成過程不是一次生成一個元素，而是透過逆轉一個逐漸破壞資訊的破壞過程來定義。標準方法是逐漸添加高斯噪聲，直到它完全壓倒訊號。
在 2019 年 4 和 2020 年 5 6 的圖像生成領域取得早期成功後，首次將此想法應用於語言的嘗試出現在 2021 年，涉及用離散過程取代連續破壞過程，以實現對分類數據的建模：多項式擴散 7、D3PM 8 和 SUNDAE 9。
當時，自迴歸的主導地位不像今天這樣確立：GPT-3 10 引起了一些關注，但「ChatGPT時刻」要到 2022 年底才會出現。那時，離散擴散似乎解決了自迴歸範式的一些實際理論缺陷，例如由於 teacher forcing 引起的暴露偏差（exposure bias），以及其在填空和約束生成任務中應用的相對困難。請注意，在之前的幾年裡，已經有一些對非自迴歸和任意順序自迴歸方法的探索 11 12（特別是對於機器翻譯 13 14），但尚未從擴散的角度進行。
2022 年，出現了幾種將連續擴散應用於語言建模的嘗試，始於 Diffusion-LM 15。這種方法以不同的方式解決了分類數據與高斯噪聲破壞之間的兼容性問題：只需將離散類別表示為連續的嵌入向量，這些向量非常適合高斯噪聲破壞。這樣，適用於圖像的高斯擴散機制就可以在不作任何更改的情況下應用。
Diffusion-LM 特別強調了這種替代生成範式在可控文本生成方面的優勢。在 2022 年的最後幾個月裡，出現了許多使用此方法變體的論文，包括 DiffuSeq 16、SSD-LM 17、Difformer 18、SeqDiffuSeq 19、GENIE 20、LD4LG 21，以及我參與的兩篇論文：自我條件嵌入擴散（SED）22 和分類數據的連續擴散（CDCD）23。
當時，這些連續方法吸引人的地方在於，它們可以從為連續擴散模型發現和開發的所有見解、工具和機制中受益，因為它完全接管了視聽生成。例如，將為連續擴散模型開發的一些取樣和蒸餾技術應用於離散擴散，通常要困難得多，甚至完全不可能。
然後，發生了一些有趣的事情：2023 年之後，該領域幾乎所有的研究都使用了離散擴散，而用於語言的連續擴散則銷聲匿跡。2025 年關於擴散語言模型的調查論文 24 中的一張圖清楚地顯示了這一點：
[圖片：擴散語言模型調查論文圖表，顯示連續方法（黃色）在 2023 年後急劇減少，離散方法（綠色）則佔據主導地位]
連續方法標記為黃色，離散方法標記為綠色。從 2023 年到 2024 年的轉變非常顯著！很難確定為什麼會發生這種情況，但我可以想到幾個潛在因素：一個是 ChatGPT 時刻，它逐漸將語言擴散研究的重點從理論優勢和優雅轉移到原始性能。現在，目標是嘗試在規模上匹配強大的自迴歸模型，甚至在特定設置中超越它們。人們似乎覺得，使用完全離散的方法更容易縮小性能差距，也許是因為它們在概念上與自迴歸更為接近。
另一個因素可能是，語言擴散模型的擴展科學已經開始被探索，而連續方法的初步觀察並不樂觀。2023 年 5 月，Gulrajani & Hashimoto 25 量化了一個基於似然的連續擴散語言模型（Plaid-1B）的訓練效率差距：效率低 64 倍。（請注意，上面的圖表將 Plaid 標記為離散，但它是一種連續方法。）
在 LLM 社群仍然非常關注訓練計算與困惑度（Chinchilla-optimality 26）的帕累托前沿時，任何訓練效率比自迴歸基準低近兩個數量級的建模方法都難以認真對待。第一個 LLaMA 27 模型，它挑戰了這種訓練效率的關注點，並主張考慮推理預算，僅在幾個月前（2023 年 2 月）剛剛發布，所以我認為社群尚未內化這種轉變是可能的。
Needless to say, this is all highly speculative. Perhaps it was just a coincidence, and discrete methods ended up having more momentum around that time purely by chance. If you have any thoughts about what could have caused the late-2023 continuous extinction event, I’d be keen to hear them in the comments!
Personally, I had stopped working on diffusion language models by that point (I got too busy building image and video generation models: Imagen and Veo , and later on, Nano Banana and Omni ), so I was just observing this evolution from the sidelines. I found it somewhat suprising, because I believed continuous diffusion has a few key advantages, like an ability to represent uncertainty at the individual token level, and a rich toolbox of sampling algorithms and tricks to draw on. Giving those up seemed like it could be a mistake, but the research community as a whole clearly figured that this was the way to go.
We will talk about what’s been happening in the diffusion language modelling space more recently in the next section , but first, I think it is useful to discuss how continuous diffusion can actually be applied to discrete data in a bit more detail. This context will be helpful to understand what might be driving recent events.
The first thing to consider is the nature of the discrete data we are trying to model. Usually when people say ‘discrete’, they actually mean categorical , i.e. the output space (at the token level) is an unstructured set, and there is no relationship (ordinal or otherwise) between the different values that each discrete variable can assume. Digital images represented as pixel grids are also discrete, technically speaking, but because the discrete values assumed by the pixel colour channels represent an underlying continuous physical signal (light intensity), we tend to simply ignore that, and treat them as continuous anyway.
Assuming we are working with categorical data, there are a few necessary ingredients to make continuous diffusion work well: an embedding strategy , a loss function , and a sensible noise schedule . In addition, there is a trick that pops up in almost every paper on this topic, which turns out to have a huge impact on performance: self-conditioning . We’ll take a closer look at each of these in turn. For brevity, I will use the acronym CDLM to refer to continuous diffusion language models going forward, and DDLM to refer to their discrete counterparts.
Modern neural networks typically have real-valued parameters and activations. Therefore, the first thing that usually happens in any neural network that processes discrete data, is embedding the discrete inputs in a continuous representation space. From that point on, the network exclusively manipulates real-valued representations. These embeddings are usually just treated as additional parameters, which can be optimised jointly with the rest of the model. It’s worth pointing out that this is also the case for DDLMs and autoregressive models – the internals of the neural networks powering these models are still continuous.
To apply continuous diffusion to discrete data, we can simply ‘lift’ the corruption process from the discrete input space into a continuous embedding space. In other words, compared to discrete diffusion, it’s just a question of changing the order of operations. Rather than applying discrete corruption followed by continuous embedding during training, we first embed the inputs and then apply continuous corruption instead.
The shape and structure of the embedding space profoundly impacts the nature of the continuous corruption process that happens within it. Various embedding strategies have been explored:
Explicit (e.g. SSD-LM 17 ): arguably the simplest approach is to use something like a one-hot representation , where every element in a vocabulary of size \(V\) is associated with a \(V\)-dimensional vector that has \(V-1\) zeros and a single one. Since a vocabulary is a set, assigning representations to the elements requires arbitrarily picking a specific ordering. This kind of embedding space can be cumbersome to work with for modern language models, because \(V\) tends to be pretty large nowadays. A potential workaround is to use compact binary patterns instead, as in Analog Bits 28 .
Pre-trained (e.g. SED 22 ): we can use a representation learning strategy to learn embeddings, and then repurpose them for use in a diffusion language model. For example, they could be borrowed from an autoregressive language model, or taken from a bidirectional language model like BERT 29 . They can also be made contextual , i.e. the embedding for a given token can depend not only on the token itself, but also on adjacent tokens, resulting in a richer embedding space 21 .
Jointly learned (e.g. CDCD 23 ): we can try to fit the embeddings together with the denoiser model in a single learning procedure, potentially letting them co-adapt.
The latter might seem like the most natural thing to do, because joint learning of the embeddings is what works for DDLMs and autoregressive LLMs. An end-to-end single-stage learning approach is also widely considered the most attractive nowadays, both from a conceptual and from a practical standpoint. But the elevated role of the embedding space in CDLMs (relative to DDLMs) means that this comes with some challenges: naive formulations are prone to embeddings collapsing or growing uncontrollably. For example, denoising error can be minimised pathologically by making all embeddings the same, so this suggests that some sort of trade-off between multiple constraints or terms in the loss function might be necessary.
There has been a fair bit of discussion in the literature about the importance of the geometry of the embedding space. It is often assumed or suggested that embedding spaces with meaningful semantic structure lend themselves better to continuous diffusion language modelling. Concretely, this means that the embeddings should be organised in such a way that a given amount of corruption creates commensurate confusion between tokens from a semantic perspective; e.g. if a small amount of noise is added to the embedding for ‘cat’, it might become indistinguishable from the embedding for ‘dog’ with the same amount of noise added, but it will still look very different from the embedding for, say, ‘umbrella’, at the same noise level.
It is still unclear to me how important this actually is, if what we care about is raw language modelling performance. This is not a factor that is usually considered in the context of DDLMs or autoregressive LLMs. It has been suggested in the literature that some strategies and learning objectives for CDLMs have a dispersive effect 30 (i.e. pushing embeddings for semantically related tokens apart, rather than together), and that this might be a bad thing, which certainly seems plausible.
As the goal of CDLMs is ultimately to generate a discrete token sequence, an unembedding strategy is also needed. Neural networks for classification tasks use a softmax nonlinearity to make predictions in a categorical space, and interpret the network outputs as probabilities (which are themselves continuous). This approach can also be used for denoisers: even if we are denoising continuous vectors, we can use the knowledge that they each represent one of a finite number of discrete vocabulary elements to constrain the predictions (in CDCD 23 , we took advantage of this observation and called it ‘score interpolation’, framing it as an alternative to ‘score matching’). In most works however, predictions are made directly in the continuous embedding space, without such constraints, and a final discretisation step is performed at the end of sampling. This is often done simply by clamping the embeddings to the nearest vocabulary element, but the procedure can also be more involved.
There is some interesting variety in the loss functions used for training CDLMs. I won’t enumerate all the options, but I do want to point out some trends. Usually, the choice of loss function is closely tied to the unembedding strategy. If the denoiser makes predictions directly in the continuous embedding space, the usual mean squared error (MSE) loss tends to be used, just like in continuous diffusion models for audiovisual data.
If the denoiser outputs probabilities across vocabulary elements, it can be trained using the categorical cross-entropy loss instead, which makes things look more similar to the autoregressive setting. Note that this approach only works with per-token embeddings and is not compatible with contextual or higher-level embeddings: predicting probabilities for every possible output is feasible at the per-token level if the vocabulary size is not too large, but not beyond that.
Another approach is to start from the maximum likelihood principle 25 , and come up with an objective that bounds the likelihood from below (in the same way that variational autoencoders 31 32 are trained). Some CDLM variants that jointly learn the embeddings and the denoiser include additional loss terms to regularise or constrain them 15 , but sometimes these constraints are handled through parameterisation instead (e.g. forcing the embeddings to be normalised vectors 23 ).
Several works have explored various ways to constrain the continuous diffusion process to the \(V\)-simplex: the space of valid categorical probability distributions across \(V\) categories 33 34 35 36 . In this setup, intermediate noisy vectors are themselves constrained to be valid probability distributions across all vocabulary elements, which also requires alternative loss functions. While it seems like a good idea in theory, in practice, this usually adds significant complexity and it doesn’t seem to be very scalable to large vocabulary sizes. Most successful applications of this idea have actually been in biology, where interesting discrete sequence modelling problems with much smaller vocabularies exist (e.g. \(V=4\) for DNA, \(V\approx22\) for amino acids).
The corruption process of a diffusion model is governed by the noise schedule, which determines the rate at which the noise level increases over the course of the process. I wrote a lot more about noise schedules for continuous diffusion models in an earlier blog post .
Ideally, the schedule is chosen so that information is destroyed gradually, allowing the generative process to be broken up into smaller subtasks that each resolve small amounts of uncertainty. A poorly chosen noise schedule results in large segments of the corruption process where nothing happens (i.e. almost no information is lost, and therefore the denoiser has nothing to learn), and some segments where a lot of information is destroyed all at once, making for a very difficult denoising task.
For CDLMs, getting the noise schedule right is especially important, as a naive strategy will almost certainly result in a very uneven corruption process. This is a direct consequence of the fact that embeddings are usually high-dimensional vectors, which represent discrete underlying categories. In that setting, meaningful corruption happens across a relatively small range of noise levels . Most noise levels either destroy almost no information about token identity (too low), or destroy almost all information (too high). It is important to avoid spending denoiser modelling capacity on those noise levels, as it will not be able to learn anything useful there.
A common strategy has been to explicitly adapt the noise schedule to the geometry of the embedding space, either offline or through online adaptation of the noise schedule during training . This creates a feedback loop, where the model predictions are used to determine which noise levels are of interest, and subsequently the distribution of noise levels sampled to corrupt training examples is adapted to focus training on precisely those noise levels.
The original inspiration for such online adaptation mechanisms was the variational diffusion models (VDM) paper 37 , which used this idea to minimise the variance of the training objective, in order to accelerate convergence. In the context of CDLMs, this approach was adapted to obtain a balanced corruption process with a focus on noise levels where the level of corruption is just right to enable learning of meaningful structure. This can be achieved by learning a schedule \(\sigma(t)\) that linearises the entropy of the denoiser predictions 23 38 (in terms of \(t\)), or the decoding error rate 39 . With \(t\) sampled uniformly, entropy linearisation ensures that each diffusion sampling step resolves the same number of bits of information.
Diffusion sampling is stateless , in the sense that the next update step in the sampling procedure only depends on the current canvas. One could consider the canvas itself to constitute a form of state, but crucially, it is always fully observed. There is no additional hidden context that the model can manipulate during sampling, which is why the sampling procedure can be (and often is) framed in terms of differential equations 40 .
This led some people to wonder if perhaps denoisers used for diffusion sampling are doing redundant work: at every sampling step, they compute the optimal denoising direction from observing the current noisy canvas, without access to their own previous predictions from earlier steps. But if the steps are small enough, the optimal denoising direction might actually be quite similar to the previous prediction, so this seems like it could be wasteful.
Self-conditioning 28 was introduced to address this: simply pass the denoiser’s previous prediction to the next step as an extra input. This allows the denoiser to learn how to modify a rough estimate, rather than having to make predictions from scratch. To train such a denoiser, the additional ‘previous prediction’ input is sometimes left blank, and sometimes provided during training by using the denoiser itself to make a prediction from scratch (hence ‘self’-conditioning). This clever mechanism ensures that the denoiser still works when no previous prediction is available, but also knows what to do with it when it is provided.
For CDLMs, it was discovered pretty quickly that self-conditioning tends to provide a huge boost in performance, and so almost all works in this space make use of it. This is in spite of the fact that it breaks the statelessness assumption built into various diffusion machinery, most notably sampling algorithms based on differential equations (ODEs and SDEs). It is fair to assume that it probably biases the modelled distribution in hard-to-understand ways, but everyone uses it anyway, because it makes such a huge difference to sample quality that it would be an act of self-sabotage not to.
Exactly why this works so well for language diffusion in particular is still unclear – especially because attempts to apply the idea for audiovisual generative modelling have been far less fruitful (Recurrent Interface Networks 41 are a notable exception). The underlying discrete structure of the output space seems to play a role in this. A recent paper by Yoo et al. 42 reanalyses diffusion with self-conditioning as an efficient approximation of a fixed-point model embedded within a diffusion model, almost like a nested for-loop. This perspective provides an explanation as to why the statefulness of denoisers with self-conditioning does not appear to be a problem in practice: it is merely a side effect of approximating the nested for-loop with a single flat loop.
To wrap up this section, I want to illustrate how these ingredients can come together in a few different ways, using some early works in the CDLM space that I contributed to. All of these date back to late 2022, because I stopped working on language after that.
Simplex diffusion 33 uses a non-Gaussian corruption process: the so-called Cox-Ingersoll-Ross (CIR) process 43 . This operates on strictly positive real values, and it was originally used to model interest rates. It comes with a built-in assumption that these interest rates cannot be negative, so as you can imagine, it lost a bit of traction for that purpose after 2008! That property does make it very well-suited to model (unnormalised) probabilities, though. We used the score-based SDE formalism (📉) with this alternative process, which (somewhat surprisingly) yields tractable, if slightly exotic formulas for all the quantities of interest. For example, the transition density is a non-central chi-squared distribution, instead of the usual Gaussian. This was a theoretical exploration during the project that later became CDCD (see below). We ended up not pursuing it further, because it seemed to scale poorly to larger vocabulary sizes.
Self-conditioned embedding diffusion 22 (SED) is built around pre-trained embeddings (📍) obtained using a BERT model, which is slightly modified to have a low-rank bottleneck, as diffusion on lower-dimensional embeddings was found to perform better. Note that even though they come from a BERT model, the embeddings themselves are per-token, not contextual. The loss function is a combination of the usual denoising MSE and a cross-entropy-based unembedding loss (📉), the noise schedule is a cosine schedule (📻 fairly standard for the time), and self-conditioning is an important ingredient (🔄 it’s in the name!).
Continuous diffusion for categorical data 23 (CDCD) is built on the principle that language diffusion would be more likely to see wider adoption if it looks as familiar as possible to existing LLM practitioners. The paper frames it as a version of BERT, but with Gaussian noise instead of masking noise. It uses standard Gaussian diffusion, but with a cross-entropy loss function (📉 score interpolation ), and with embeddings learnt on the fly, jointly with the denoiser (📍). As this makes the embeddings prone to uncontrollable growth, a normalisation layer is used to force them to have unit norm. It also relies heavily on self-conditioning to achieve good performance (🔄). Another key performance factor is the adaptive noise schedule based on an entropy linearisation heuristic (📻 time warping ), which ensures that both training and sampling spend more time and capacity on the noise levels that matter most.
To my own delight, many of the CDCD ingredients have become fairly mainstream in modern CDLM works (several of which I’ll discuss in the next section). Adaptive schedules feature frequently, and they often use some sort of linearisation heuristic. The score interpolation strategy that originally enabled cross-entropy-based training of continuous denoisers has been rederived in a more modern setting (i.e. flow matching 44 and flow maps 45 ), and given a stronger theoretical underpinning. Self-conditioning is now ubiquitous.
With all of that in mind, let’s pick up where we left off at the end of the first section , and take a look at what’s been going on with CDLMs more recently.
After 2023, this space was very quiet for a long time as people focused on discrete diffusion. Two strategies for discrete corruption are commonly used: masked discrete diffusion corrupts tokens by gradually replacing all of them with mask tokens, until the sequence is fully masked. uniform-state discrete diffusion corrupts tokens by replacing them with random tokens instead, until the sequence is fully randomised. The former approach has a single deterministic absorbing end state (fully masked), whereas the end state of the latter is that all possible token sequences are equally likely (uniform categorical distribution).
In the second half of 2025, people started trying to bring back some continuous flavour in the form of hybrid methods , combining discrete and continuous approaches in various ways to try and get the best of both worlds. This was followed in 2026 by a full-on resurgence of continuous methods.
Sahoo et al. 46 started off this trend by observing a close connection between Gaussian continuous diffusion and uniform-state discrete diffusion. They found that mapping continuous noisy intermediate states to discrete states using the \(\arg \max\) operator also implicitly turns the Gaussian corruption process into a uniform-state corruption process. They called this relationship the diffusion duality , and used it to apply consistency distillation 47 to discrete diffusion models, as well as for training loss variance reduction.
CADD 48 , CCDD 49 and CANDI 50 all suggest different ways to combine discrete and continuous corruption into a single process. CADD uses continuous intermediate representations to augment masked diffusion, in order to ensure that information is lost in a more gradual fashion. CCDD uses joint diffusion over discrete and continuous representations simultaneously, to tap into the increased expressivity of continuous diffusion, while avoiding the challenge of decoding continuous embeddings back into discrete tokens.
CANDI instead tries to address a scaling issue with continuous diffusion for discrete data, which they call temporal dissonance : for high-dimensional vocabularies, the discrete identity of individual tokens decays quickly as the corruption process progresses, but their relative rank among all posibilities decreases much more slowly. By the time there is anything interesting to learn about the semantic structure of the continuous embedding space, all token identities have already been corrupted, and the model will have a really difficult time learning about the conditional relationships between tokens as a result. They identify this as a key problem holding back continuous methods, and propose to address it by using discrete masking and applying Gaussian corruption only to the masked positions, thereby decoupling these two kinds of corruption.
These works all appeared within a few months of each other, making it almost seem like a coordinated effort to rehabilitate continuous diffusion for discrete data, while still sticking closely to the dominant discrete paradigm. Not long after this, the purely continuous approach also saw a revival, as we will discuss next.
2025 had also seen the development and rapid adoption of flow maps 45 51 , the subject matter of my previous blog post . A flow map is essentially the integral of a diffusion model. In a diffusion model, a denoiser is learnt, which can be used to move through input space from noise to data by repeatedly taking small steps in the predicted denoising direction. Flow maps try to do this in one go instead, or at least, in as few steps as possible. To achieve this, a network is trained to directly approximate the output of the diffusion sampling procedure. (I’m deliberately cutting some corners here for brevity’s sake, but that is the gist of it.)
Although step distillation of diffusion models had already been a fruitful research topic long before that, the development of such a powerful framework seemed to inspire several groups of researchers to revisit continuous methods for language modelling, in hopes of bringing the benefits of the framework to this class of models as well. Early 2026 saw the appearance of three closely related works in rapid succession: Categorical Flow Maps 52 , Flow Map Language Models 39 and Discrete Flow Maps 53 . All three extend flow maps to the discrete categorical setting using explicit one-hot embeddings (📍) and cross-entropy-based loss functions (📉), bringing their application to language modelling within reach.
In the spring of 2026, this was followed by something of a Cambrian explosion in the space of CDLMs. A series of papers appeared that revisited and extended various recipes:
LangFlow 54 , Spherical flows 55 and Hyperspherical flows 56 build on CDCD 23 , making use of jointly learned normalised embeddings (📍), the cross-entropy loss (📉) and adaptive noise schedules (📻). The latter two works constrain the corruption process to the sphere (we tried a naive variant of this in CDCD and called it ‘renormalisation’, but that didn’t work very well).
Latent diffusion language models (LDLM) 57 and Embedded language flows (ELF) 58 follow the design of Diffusion-LM 15 and SED 22 , applying the standard continuous diffusion recipe in an appropriately chosen embedding space. A major difference is that they make use of contextual embeddings, rather than per-token ones (📍 following LD4LG 21 ). They are jointly learnt with the denoiser for LDLM, whereas ELF (primarily) uses pre-trained and frozen embeddings.
Continuous bitstream diffusion (CoBit) 59 uses explicit embeddings in the form of bit sequences (📍), applying and extending the Analog Bits 28 approach to language.
RePlaid 30 revisits Plaid 25 and modernises its likelihood-based approach (📉) by drawing from recent DDLMs.
Compared to their predecessors, these works feature a modernised framing, improved implementations, new theoretical insights, updated evaluation methodologies and increased scale. Several of them argue that the previous consensus about discrete diffusion having the edge is incorrect . RePlaid and LangFlow even make the opposite claim in their respective paper titles: ‘Continuous Diffusion Scales Competitively with Discrete Diffusion for Language’, ‘Continuous Diffusion Rivals Discrete in Language Modeling’ .
The revival of CDLMs is very much ongoing, so it is probably a bit too early for a historical perspective that tries to fully explain the back-and-forth between discrete and continuous methods over the past five years. Nevertheless, I want to point out a few trends that have likely influenced it.
First of all, continuous methods have gotten simpler and easier to use over the years: earlier perspectives required understanding score matching 4 , deep latent variable models 6 or differential equations 40 ; modern explanations rely mostly on basic concepts like linear interpolation between data and noise 44 . This has lowered the barrier of entry for practitioners in adjacent fields to also explore these methods.
Interest in alternative modelling paradigms for language beyond autoregression has also increased overall, thanks to the success of DDLMs and the quest to explore novel substrates for reasoning. The potential pay-off of finding the next big thing has only increased as large language models have become big business: making language models faster and more flexible has become highly economically valuable .
Attempts to decrease the number of steps required to sample from DDLMs tend to hit a wall: simultaneously sampled tokens are assumed to be conditionally independent given previously sampled tokens. In the limit of single-step sampling, this means all tokens are necessarily sampled independently, and the models are then fundamentally unable to capture any correlations between them. This makes step distillation challenging, and overcoming that problem might require introducing significant additional complexity.
Continuous methods side-step this issue completely: trajectory-based step distillation methods (like flow map methods) enable even single-step models to capture all correlations, at least in theory – in practice, the limited capacity of the models still makes this challenging to do in just one step. Nevertheless, I think it is fair to say that few-step sampling comes much more naturally to the continuous setting. This distillability advantage is probably the main reason why CDLMs are back with a vengeance today.
Step distillation doesn’t just enable faster sampling: it also unlocks possibilities for reward-based steering and fine-tuning that were challenging to achieve with diffusion-based language models before (see this section in my previous blog post). Given the major role that post-training plays in the success of modern LLMs, this is also an important consideration.
Whether so-called flow map language models will displace the current status quo wholesale remains to be seen, but I fully expect them to continue to gain traction, and efforts to scale them up are underway 60 . I strongly recommend reading Floor Eijkelboom’s deep dive on flow-based language generation for a more technical treatment of this topic. Jiaming Song also has a nice write-up on the role of diffusion and flow maps in language modelling.
It is interesting to ponder how the dominant position of continuous diffusion models for audiovisual generation is motivated by almost entirely different reasons. I have previously written extensively about the spectral perspective on diffusion , and the link between noise levels and feature scales in the visual domain. I believe the fact that we can manipulate the diffusion loss to emphasise perceptually relevant content is a key reason for their success, but that clearly does not apply to language modelling at all.
One commonly heard argument in favour of CDLMs is that they make multimodal integration easier: we can use continuous diffusion across all modalities and combine them into a single model. While that is true, I think it misses the point a little bit. The challenge of building multimodal generative models is not so much about bridging multiple modelling paradigms, which is actually not that difficult (see e.g. Diffusion Forcing 61 , Transfusion 62 ). A more pertinent challenge is the semantic gap that exists between language representations and representations of perceptual signals used in diffusion models (i.e. latents or patches of pixels): language tokens are semantically abstract, audiovisual tokens are not. I will probably have more to say about that in a future blog post. For now, all I have is a spicy tweet 🌶️:
[Tweet: In a multimodal context, even the discrete/continuous divide is a distraction. The real challenge is bridging the semantic gap between inherently high-level language tokens, and the very low-level representations we tend to use for perceptual signals. (I couldn't resist😆) https://t.co/0or588gBV5]
Is this the end of the line for discrete diffusion, then? That seems rather unlikely. Dimitri von Rütte made the case that diffusion language models are the future a while back (uniform-state discrete models in particular). More recently, Volodymyr Kuleshov and colleagues shared a post describing the building blocks of DDLMs, and Junbo Zhao wrote a blog post about their move into the mainstream. A lot of people seem to be bullish enough about the approach to blog about it!
The recent release of DiffusionGemma 63 , an open-weights uniform-state DDLM developed by my colleagues at Google DeepMind, as well as NVIDIA’s Nemotron Diffusion 64 , also significantly increased awareness that autoregression is not the only game in town.
Step distillation of DDLMs is not a complete impossibility either: methods such as discrete moment matching distillation (D-MMD) 65 and inverse-distilled diffusion language models (IDLM) 66 show that some approaches can be ported over from the continuous to the discrete setting.
It is worth noting that displacing autoregression entirely is not the only way for diffusion language models to be successful. They can coexist, sometimes even within the same system. A common strategy to accelerate sampling from autoregressive language models is speculative decoding, where a faster draft model is used to predict multiple tokens at a time, which can then be verified in parallel by the autoregressive model. Discrete diffusion is increasingly being used for drafting in this context (e.g. DFlash 67 ).
A different perspective on the same idea is that autoregressive verification can be used to mitigate the impact of independence assumptions in few-step discrete diffusion sampling 68 . DMax 69 uses a combination of continuous relaxations and post-training to reduce the impact of these independence assumptions instead.
In the meantime, some recent works have continued to explore hybrid continuous-discrete approaches. Sticky Jump Diffusions 70 are an attempt to create a unified view of hybrid methods, building on the SDE formalism, with both discrete masked diffusion and continuous diffusion as special cases. Posterior Refinement 71 wraps continuous diffusion within masked diffusion. This creates a nested sampling loop, where the inner loop can be distilled down to very few steps using flow map methods . This results in a form of masked diffusion where each step can also capture correlations between simultaneously unmasked tokens (unlike standard masked diffusion, which assumes their conditional independence).
The current trend of diffusion language models moving into the mainstream shows no signs of slowing down. Whether discrete or continuous methods will come to dominate is hard to predict – perhaps they will coexist, alongside autoregression. As the LLM community moved on from the Chinchilla perspective (focusing exclusively on training efficiency), alternative modelling paradigms have gradually received more attention, which diffusion language models are undoubtedly benefiting from. In the longer term, another aspect of diffusion models may gain importance: their increased data efficiency relative to autoregression 72 .
In the meantime, I believe it is crucial for the research community to work to address a common weakness of diffusion language modelling papers: the evaluation methodology . Because of their flexible sampling procedure, diffusion models are particularly amenable to tuning the trade-off between quality and diversity at sampling time. If this trade-off is not carefully quantified and accounted for, this can lead to certain models appearing to be significantly better than others, even if they merely represent different points along this trade-off. Because the evaluation methodology is not currently standardised, different papers use different approaches and results can be unintentionally misleading.
Moreover, because many formulations do not readily admit the estimation of likelihoods or perplexities under the models themselves, surrogate autoregressive models are often used to measure these instead (referred to as generative perplexity , GenPPL), which biases the evaluation towards the capabilities and weaknesses of the surrogates used.
Patrick Pynadath and colleagues suggest quantifying the trade-off by looking at generative frontiers (i.e. plotting perplexity vs. entropy) in a recent blog post . Sam Acquaviva also has a blog post identifying several diffusion language model evaluation issues and potential fixes. Franca and Tong 73 demonstrate just how easy it is to game GenPPL as a metric, and argue that it should not be used even when entropies are matched. Metrics such as MAUVE 74 were proposed to try and address the challenge of evaluating open-ended text generation, but ultimately still rely on pre-trained autoregressive models.
A research direction that continues to capture people’s imagination is latent diffusion for language : learning a higher-level and potentially more coarse-grained continuous representation for language that is easy to model with vanilla continuous diffusion. The main challenge here continues to be learning the latent space itself, not so much the diffusion part. As previously discussed in my blog post on latent diffusion , language is a very different beast compared to perceptual signals, and representation learning techniques that work well for the latter might completely fail for the former.
Aside from LD4LG 21 and LDLM 57 (previously mentioned), there has been a steady stream of work on learning higher-level language representations at the token, phrase, sentence or paragraph level, including Time Control 75 , PLANNER 76 , Large Concept Models 77 , Segment-level Diffusion 78 , LaDiR 79 , Latent Thought Flows 80 and AURORA-LM 81 . Mapping out this space would lead us too far, but this line of work cannot go unmentioned when talking about CDLMs, even if it is not the focus of this post.
I wanted to write a quick note about the recent resurgence of CDLMs, given my earlier work in the space, my fondness for the idea, and my disappointment when DDLMs seemed to take over completely after 2023. Inevitably, it turned into an essay and a historical account of language diffusion research over the past five years – I just can’t seem to help myself 🤷. Some key takeaways to wrap up:
I also want to highlight a few upcoming diffusion language modelling workshops :
The submission deadline for the NeurIPS workshops is in a few days! Thanks for reading, and as usual, please feel free to share your thoughts in the comments, on Twitter , or via email.
Disclosure regarding the use of AI in producing this blog post : I want to write in my own voice, and I want to respect everyone who takes the time to read what I write. Therefore, you will not find any passages or sentences in this post that are fully AI-generated. (Even the em dashes are all mine!) That said, I do occasionally consult AI when considering a particular turn of phrase, or to help me find the best wording (like a souped-up version of thesaurus.com). I primarily use it to help me understand papers and the relationship between them, and sometimes to create images and diagrams. AI was extensively used in the making of this blog post, but the prose is entirely ‘artisanal intelligence’. That is the level of AI involvement I am currently comfortable with.
If you would like to cite this post in an academic context, you can use this BibTeX snippet:
Many thanks to James Thornton, Oliver Wang, Sheel Shah, Jinwoo Kim, Justin Deschenaux, Patrick Pynadath, Oscar Davis, Luca Ambrogioni and Zhengyang Geng for sharing their thoughts and insights on this and many other topics. I’d also like to thank the organisers, other speakers and participants of the wonderful EEML 2026 summer school in Cetinje, Montenegro, and the ICML 2026 diffusion circle crew. As usual, thanks to my colleagues at Google DeepMind and the wider research community.
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, “ Attention is All you Need ”, Advances in neural information processing systems 30 (NeurIPS), 2017. ↩
Williams, Zipser, “ A learning algorithm for continually running fully recu