Value-gradient guidance is least accurate at high noise, where a sample commits to a mode. VaLoP reads the value one step ahead and pulls its gradient back, and VaLoP-Fast replaces the pullback with a lightweight head.
Reward guidance steers a pretrained diffusion or flow model toward images or actions that a reward model prefers, at sampling time and without retraining the model for each reward. It follows the gradient of a value function, the expected reward given the noisy state. The two common estimates of this gradient, gradients through a one-step denoised estimate and learned value models, are least accurate at high noise, where the coarse structure of the sample is decided. Interestingly, we find that both errors are posterior covariances, which grow with the uncertainty about the clean sample, and hence at higher noise.
To reduce them, we evaluate the value at a less noisy state, one sampler step ahead, where this uncertainty is smaller, and pull its gradient back to the current state. This lookahead places both estimators at the two ends of one family and reveals that an intermediate lookahead requires one additional term, a correction through the generator. We therefore propose Value Gradient Lookahead and Pullback (VaLoP), which computes this correction with one vector-Jacobian product, and VaLoP-Fast, which amortizes it with a small head that, for text-to-image models, is trained with a single reward and reused for the others.
Across various models, rewards, and benchmark datasets, both robustly outperform existing value-gradient guidance on the target reward, VaLoP-Fast at about the cost of learned-value guidance. Both extend to critic-guided robot policies, where VaLoP-Fast attains the highest success rates on LIBERO and RoboCasa.
Errors grow with noise
The errors of both common value-gradient estimates, DPS and a learned value model (StitchVM), are posterior covariances that grow with noise. VaLoP reads the value one step ahead, where the clean sample is less uncertain, and pulls the gradient back.
At scale: one guided step on SD3.5-Medium
StitchVM's HPSv2 gain per step drops by more than half at high noise. Read one step ahead, the same value model gains 2.1× as much at t = 0.6–0.8.
Mean ± s.d., as in the paper's Fig. 3c.
In a controlled toy: where the samples end up
Colour marks the mode each sample ends in. Exact value guidance sends 68% of the samples to the rewarded mode, DPS 48%, StitchVM 43%, and VaLoP 55% at the lowest MMD² of the three.
Look ahead, then pull back
Look ahead
One sampler step to a less noisy state, at s = 0.6t.
Read the value there
The value model is far more accurate here than at zt. For robot policies, a critic read at the denoised action plays its role.
Pull it back
ξt costs one VJP through the generator in VaLoP and one head pass in VaLoP-Fast. For robot policies, the head predicts gt itself, with no critic.
Guide the current state
Step toward the rewarded mode, with the generator and value model frozen.
At s = t this is learned-value guidance, and at s = 0 the DPS form. In between, one extra term appears, the correction through the generator.
Switch and see
Drag or press Flip to compare the unguided model with ours under the same seed. ✓ and ✗ are GenEval verdicts.
Base
VaLoP-Fast
“a person and a bear”
Ours is VaLoP-Fast (VaLoP for the microwaves). The unguided model and StitchVM fail all eight prompts.
Same start, different finish
A task-specific critic guides frozen VLA policies, as in AGF. Each row replays one episode from the same initial state, and a panel freezes when its episode ends. Click a row to enlarge it.
LIBERO
SmolVLApick up the black bowl next to the cookie box and place it on the plateLIBERO-Spatial
open the middle drawer of the cabinetLIBERO-Goal
RoboCasa
π0.5Open the left drawer.RoboCasa365 atomic task
Close the toaster oven door.RoboCasa365 atomic task
Faster and lighter
VaLoP-Fast skips the generator backward pass, so it costs about as much as learned-value guidance.
Image generation
Reuse and robots
Times and memory from Tab. 3 on one H100, with StitchVM run in the same implementation as ours.
Better target reward, higher success
Bars start at zero. Values from Tab. 1 (SD3.5-Medium, SD3.5-Large and FLUX.1-dev on DrawBench) and Tab. 2 of the paper.
BibTeX
@article{kim2026valop,
title = {VaLoP: Value Gradient Lookahead and Pullback for Reward-Guided Diffusion},
author = {Kim, Kwanyoung},
journal = {arXiv preprint arXiv:2610.35607},
year = {2026}
}