![[2602.18695] Insertion Based Sequence Generation with Learnable Order Dynamics](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18695] Insertion Based Sequence Generation with Learnable Order Dynamics
Summary
The paper discusses a novel approach to sequence generation using insertion models with learnable order dynamics, enhancing flexibility and quality in generating variable-length sequences.
Why It Matters
This research addresses the limitations of autoregressive models in sequence generation by introducing a method that allows for more flexible and efficient learning. The findings could significantly impact fields such as natural language processing and drug discovery, where generating high-quality sequences is crucial.
Key Takeaways
- Insertion models offer greater flexibility than autoregressive models for sequence generation.
- Trainable order dynamics improve the efficiency and quality of generated sequences.
- Empirical results show that learned order dynamics enhance the generation of valid small molecules.
Computer Science > Machine Learning arXiv:2602.18695 (cs) [Submitted on 21 Feb 2026] Title:Insertion Based Sequence Generation with Learnable Order Dynamics Authors:Dhruvesh Patel, Benjamin Rozonoyer, Gaurav Pandey, Tahira Naseem, Ramón Fernandez Astudillo, Andrew McCallum View a PDF of the paper titled Insertion Based Sequence Generation with Learnable Order Dynamics, by Dhruvesh Patel and 5 other authors View PDF HTML (experimental) Abstract:In many domains generating variable length sequences through insertions provides greater flexibility over autoregressive models. However, the action space of insertion models is much larger than that of autoregressive models (ARMs) making the learning challenging. To address this, we incorporate trainable order dynamics into the target rates for discrete flow matching, and show that with suitable choices of parameterizations, joint training of the target order dynamics and the generator is tractable without the need for numerical simulation. As the generative insertion model, we use a variable length masked diffusion model, which generates by inserting and filling mask tokens. On graph traversal tasks for which a locally optimal insertion order is known, we explore the choices of parameterization empirically and demonstrate the trade-offs between flexibility, training stability and generation quality. On de novo small molecule generation, we find that the learned order dynamics leads to an increase in the number of valid molecules ge...
