ELF: Embedded Language Flows
Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li + 3 more
TLDR
ELF proposes a continuous diffusion model for language, leveraging flow matching in embedding space to achieve superior generation quality with fewer steps.
Key contributions
- Introduces Embedded Language Flows (ELF), a continuous diffusion model for language based on Flow Matching.
- Operates predominantly in continuous embedding space, mapping to discrete tokens only at the final step.
- Enables straightforward adaptation of established image-domain techniques like classifier-free guidance (CFG).
- Substantially outperforms leading discrete and continuous diffusion language models in quality and efficiency.
Why it matters
Current diffusion language models primarily operate over discrete tokens, limiting their effectiveness. ELF offers a novel continuous approach that simplifies adapting established techniques from image generation. This work paves the way for more effective and efficient continuous language generation.
Original Abstract
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.