iADD: Improving Alignment and Diversity
in Diffusion Policy Optimization

Key facts

Abstract

Reinforcement learning based post-training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimization do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design. We analyze the theoretical framework and mathematically demonstrate that only-latter-timestep updates of a diffusion model may be harmful for diversity, contrary to the conclusions presented in previous work. Additionally, we propose an incremental Feynman–Kac training procedure, grounded in strong theoretical foundations, to achieve the best-yet alignment–diversity tradeoffs. We perform extensive experiments and compare our method against related diffusion policy optimization approaches on three different tasks, and provide strong ablations for each component, validating performance gains in both alignment and diversity.

Method

Interactive: FK branch & resample

k particles branch from one prefix; low-weight ones are resampled away.

Interactive: sparse incremental curriculum

Only a growing subset of timesteps gets gradient updates; early steps matter most.

Results

Alignment, diversity and rarity across tasks

Each method evaluated at its best reward–diversity operating point. iADD gets the strongest reward–diversity AUC and rarity across all three tasks while matching or beating DDPO / B²-DiffuRL on raw reward.

iADD's trajectory guidance and sparse curriculum are complementary to the underlying policy optimizer: combining them with GRPO raises CLIPScore and rarity further while improving LPIPS diversity.

Qualitative results

FAQ: RL fine-tuning of diffusion models without losing diversity

iADD is a diffusion policy optimization method for fine-tuning diffusion models with reinforcement learning (RLHF for diffusion). It builds on DDPO, DPOK and B2-DiffuRL, brings Feynman-Kac (FK) steering / sequential Monte Carlo ideas into training, and combines with GRPO methods such as DanceGRPO and BranchGRPO.

What is iADD?

iADD is a reinforcement learning fine-tuning method for diffusion models that combines an incremental, sparse timestep-update curriculum with discrete-time Feynman-Kac branching during training, to improve alignment and diversity together.

Why does RL fine-tuning of diffusion models cause mode collapse?

Optimizing a reward over all denoising steps pushes the model away from its prior and toward a few high-reward modes. iADD's sparse updates preserve more of the prior's probability volume (log-det -25,963 sparse vs -27,568 dense).

Why not update only late timesteps like B2-DiffuRL?

iADD's Proposition 1 shows uniform timestep sampling gives more generative reach than late-only updates; in ablations, updating any N timesteps beats updating the last N on Inception Score.

How is iADD different from Feynman-Kac (FK) steering?

FK steering applies Feynman-Kac guidance at inference time. iADD uses it during training: low-potential branches are pruned for alignment and promising ones are branched for diversity, so no test-time scaling is needed.

DDPO vs iADD: what changes?

DDPO fine-tunes the full denoising trajectory. iADD updates sparse, incrementally chosen timesteps and adds training-time FK branch/resample, reaching 66.85% rarity vs 24.81% for DDPO on images.

Does iADD work with GRPO (DanceGRPO)?

Yes. iADD+GRPO raises CLIPScore from 0.3886 (DanceGRPO) to 0.4126 and rarity from 61.67% to 88.33%.

Does iADD reduce reward hacking?

In 3D scenes it has fewer collisions at matched reward than DDPO and B2-DiffuRL; on images, baselines drift to cartoonish, over-saturated outputs.

Does iADD need preference data or extra inference compute?

No. It optimizes a reward function (no human preference pairs, unlike Diffusion-DPO) and adds no test-time scaling.

What tasks was iADD tested on?

Text-to-image generation with Stable Diffusion v1.4 + LoRA and a CLIPScore reward, vanishing-point correction, and 3D indoor scene synthesis (MiDiffusion on 3D-FRONT).

@misc{neupane2026iadd,
  title   = {iADD: Improving Alignment and Diversity in Diffusion Policy Optimization},
  author  = {Neupane, Ashok Prasad and Adhikari, Saugat and Paudel, Pramish
             and Chhatkuli, Ajad and Paudel, Danda Pani},
  year    = {2026},
  eprint  = {2610.01789},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}