Abstract

Insertion-based diffusion language models such as Insertion Transformer and InsNet are promising alternatives to autoregressive models, with better inference-time efficiency and controllability. Yet their training-time scalability remains limited by computational inefficiency and obsolete model designs. We address this with InsNeXt, an insertion-based diffusion language model architecture that integrates recent language-model system advances for scalable pre-training. We scale InsNeXt from 154M to 0.6B parameters with a context window of 4096 by combining sentence-level and document-level pre-training, improving context encoding while preserving bidirectional context access. We also propose a decoding-time mechanism that sparsely introduces bidirectional context re-encoding, leveraging bidirectional reception with autoregressive-like computational efficiency. We evaluate pre-trained InsNeXt models on representation learning, commonsense reasoning, and controllable generation. InsNeXt matches or outperforms similar-sized autoregressive models, serving as both a solid representation learner and a powerful controllable insertion-based diffusion generator.

Date

2026

Authors

Sidi Lu, Jacky Dai, Xuezhe Ma, Nanyun Peng

Conference

Non-Autoregressive Language Models for Fast and Flexible Text Generation