Reward Vectors Feed Search

If VPO is right, the next AI training advantage is not a bigger scalar reward model but ownership of decomposed reward vectors that keep useful alternatives alive for search.

A May 2026 arXiv preprint, Vector Policy Optimization, attacks a deployment mismatch: LLMs are increasingly used as generators inside best-of-k and evolutionary search loops, while post-training still often optimizes a fixed scalar reward that narrows the response distribution. VPO replaces scalar convergence with a set-level objective: generate multiple candidates, sample reward weightings over a vector-valued reward, and reward coverage of the Pareto front rather than collapse to one mode ^[1].

The asymmetric signal is not “diversity is good.” It is that verifier-legible diversity becomes inventory: a model trained to preserve different reward tradeoffs may be more valuable inside search systems than one trained to maximize a single average score ^{[1] [2]}.