LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
Preference alignment methods built for autoregressive models — RLHF, DPO, PPO — all derive gradients from sequential per-token log-probabilities. Masked diffusion LMs don't have those: their likelihood is defined over masking schedules across all positions simultaneously. LLaDA 1.5 solves this cleanly, and the gains across math, code, and alignment benchmarks are some of the most convincing reported for dLLM post-training.
LLaDA 1.5 identifies the root problem in dLLM preference optimization as high variance in the masked ELBO gradient — not bias, which can be managed, but variance, which causes training instability and poor reward generalisation. VRPO derives closed-form variance bounds for two complementary strategies — optimal Monte Carlo budget allocation across denoising steps and antithetic sampling — and applies both in combination. The resulting estimator is unbiased relative to a naïve Monte Carlo baseline while achieving significantly lower gradient variance. Applied to LLaDA-8B, VRPO achieves gains on every benchmark tested, with the strongest improvements on alignment tasks (IFEval, Arena-Hard) where training signal quality matters most. The model checkpoint is publicly available at HuggingFace (GSAI-ML/LLaDA-1.5).