<Technical Deep Dive> How Google DeepMind Achieved 1,000x Data Efficiency in RLHF
Seyed Mohammad Asghari
Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models with human preferences. But it has a dirty secret: it is extraordinarily wasteful with human data. A new paper from Google DeepMind's Efficient Agent Team tackles this head-on, demonstrating that a combination of online learning, epistemic uncertainty modeling, and information-directed exploration can yield a projected 1,000x improvement in data efficiency over standard offline RLHF. These are striking results that, if they hold up at scale with real human raters, could fundamentally change how we think about the economics and logistics of post-training.
The Problem: Offline RLHF Doesn't Scale Well
The standard RLHF pipeline is well understood: collect a large dataset of human preference comparisons, train a reward model on those comparisons, then optimize a language model policy against that frozen reward model. This is the "offline" approach, and it has a fundamental flaw. Because all preference data is collected upfront using the base (pre-RLHF) policy, the responses being compared may have little relevance to the regions of output space that the improving policy actually occupies. You are asking humans to judge outputs the model would never actually produce after a few rounds of optimization. This is a classic off-policy problem.
Recent work has even suggested that offline RLHF exhibits diminishing returns: throwing more preference data at the problem yields disappointingly small gains. The paper frames this as a question of scaling laws. When you plot win rate against the number of preference labels on a log scale, offline RLHF produces a relatively flat curve. The central claim of this paper is that this is not an inherent limitation of RLHF, but a consequence of how data is collected and used.
Four Algorithms on a Spectrum
The paper compares four algorithms, each progressively more sophisticated in how they interact with the data collection process. All share the same base architecture (a Gemma 9B model) and similar update rules, so the comparison isolates the effect of the data collection strategy.
Offline RLHF is the baseline. Collect all T batches of preference data using the initial policy, train a reward model on the full dataset, then optimize the policy against that frozen reward model. This is the standard two-stage pipeline most practitioners are familiar with.
Periodic RLHF introduces on-policy data collection but in a chunked fashion. Every τ batches (400 in their experiments), the policy is updated, and subsequent data is collected using the new policy. The reward model and policy are retrained from scratch each period using all accumulated data. This is known to help, but it is computationally expensive since you are retraining from scratch each cycle.
Online RLHF takes this to its logical extreme: after every single batch of 64 prompts, both the reward model and the policy are incrementally updated. There is no retraining from scratch. The reward model is adjusted via a gradient step on the Bradley-Terry log-likelihood, and the policy is updated via a variant of REINFORCE where the reinforcement signal is the reward model's predicted preference probability minus 0.5. This makes updates fully on-policy at every step, but it introduces a well-known instability problem: online RLHF methods tend to "tank" — performance climbs for a while, then collapses.
Information-Directed Exploration builds on online RLHF but adds an epistemic neural network (ENN) that models uncertainty in the reward function. Instead of randomly selecting which pair of responses to show the human rater, this algorithm selects the pair that maximizes the variance in predicted choice probability across ensemble members. In other words, it asks humans to judge the cases where the reward model is most confused.
The Affirmative Nudge: A Simple Fix for a Hard Problem
One of the most interesting contributions here is also one of the simplest. Online RLHF has been known to suffer from "tanking" — a phenomenon where the policy improves for a while and then performance suddenly degrades. Previous solutions involved either checkpointing and rolling back, or reducing the learning rate. Both are unsatisfying: the first requires knowing when things went wrong (which is hard to determine in real time), and the second slows learning across the board.
The authors' fix is to add a small positive constant ε (the "affirmative nudge") to the reinforcement signal. In the standard formulation, the policy gradient for a response Y is weighted by p(Y ≻ Y'|X) - 1/2, meaning the model gets positive reinforcement when the reward model thinks Y is better than Y', and negative reinforcement otherwise. The nudge shifts this to p(Y ≻ Y'|X) - 1/2 + ε, so even a response that is predicted to be slightly worse than its counterpart still receives a small positive training signal. The intuition is that this prevents the destructive feedback loop where the model starts producing degenerate outputs that the reward model then heavily penalizes, causing further degeneration. It acts as a form of implicit regularization that keeps the policy from straying too far into pathological territory. Empirically, it eliminates tanking entirely, allowing the online algorithm to train stably over the full data budget.
Epistemic Neural Networks and Information-Directed Exploration
The biggest performance gains come from the information-directed exploration component, which rests on a reward model architecture called an epistemic neural network (ENN). The standard reward model takes a prompt X and response Y and outputs a scalar reward r(Y|X). The ENN extends this by adding an "epistemic index" Z, producing r(Y|X, Z). an ensemble of MLP heads. There is one "point estimate" MLP head (mlp0) with two hidden layers of width 1024, plus 100 "prior" networks (width 256, never updated during training) and 100 "differential" networks (width 1024, updated during training). When Z=0, inference goes through the point estimate head alone. When Z=k for k in 1..100, the outputs of the k-th prior and differential networks are added together to produce the reward. This is the randomized prior function approach from Osband et al., and it is a computationally efficient way to approximate Bayesian uncertainty. Crucially, the additional parameters from all 200 ensemble heads amount to less than 5% overhead on top of the 9-billion-parameter backbone.
The query selection strategy is then straightforward: for each prompt, sample 16 responses from the current policy (using top-5 sampling for diversity), compute the variance of the predicted choice probability across all 100 ensemble particles for every possible pair of responses, and select the pair with maximum variance. This is the pair where a human judgment will be most informative — where the ensemble members most disagree about which response a human would prefer. This is a practical instantiation of information-directed sampling (IDS), a principled exploration framework from the bandit literature.
The Update Mechanics in Detail
It is worth digging into the policy update rule, because it reveals several careful design choices. The policy gradient for a given prompt X and response pair (Y, Y') is a REINFORCE-style gradient weighted by the reward model's preference prediction, plus a KL-divergence regularization term that penalizes the policy for drifting too far from an exponential moving average anchor. This anchor is a running average of past policy parameters, not the original SFT checkpoint. Using a moving anchor rather than the fixed SFT baseline means the regularization adapts as the policy improves, rather than forever pulling it back toward its pre-RLHF state.
The online algorithms also use a clever response pair selection strategy for policy updates. After receiving the human's choice for a batch, the algorithm constructs four response pairs per prompt for the policy gradient: the queried pair in both orderings, plus the highest-reward and lowest-reward responses (according to the updated reward model) paired together in both orderings. A second batch of 64 fresh prompts is then used with reward-based pair selection only (no human queries), effectively squeezing more learning from the reward model between human feedback rounds. This means the policy gets eight gradient signals per human comparison — a form of offline augmentation within the online loop.
Experimental Setup and Results
The experiments use Gemma 9B as the base language model and simulate human feedback using a Gemini 1.5 Pro-based reward model trained on real human preference data. Using a simulator rather than live humans allows running at scale (200K+ training prompts) with controlled, reproducible comparisons. The simulator computes rewards and converts them to preference probabilities via the Bradley-Terry model, then samples a binary choice. The authors argue this setup is actually a conservative test of their approach: since the simulated preferences come from a model far more capable than the 9B Gemma policy, the preference structure is complex enough to be representative of real human judgments.
Performance is measured as win rate against the base (SFT-only) policy on 1K held-out prompts, where both the trained policy and baseline generate top-1 (greedy) responses, and the simulator produces a preference probability. A win rate of 0.5 means no improvement; higher is better.
The headline results are compelling. When plotted on a log scale, the win rate curves for each algorithm reveal starkly different scaling behaviors. Offline RLHF shows a slow, sublinear climb. Periodic RLHF does better. Online RLHF does substantially better still. And information-directed exploration dramatically outperforms them all, reaching at roughly 20K labels the same win rate that offline RLHF only achieves at 200K+ labels. That is the demonstrated 10x gain.
The 1,000x claim comes from extrapolation. The authors fit power-law scaling curves of the form w(n) = 1 - 0.5(n/a)^(-b) to each algorithm's win rate trajectory and project forward. Because the two curves have different exponents b, the gap between them widens on a log scale. At 1M labels, the projected gap corresponds to roughly a 1,000x difference in data efficiency. This is an extrapolation, not a measured result, but the functional form fits the observed data well, and the qualitative trend — a widening gap — is clearly visible in the empirical curves.
Why It Works: Decomposing the Gains
The gains come from three distinct sources, and the paper's ablation structure lets us roughly attribute them.
First, on-policy data collection (going from offline to periodic to online) accounts for a large share. When the policy generates its own responses for evaluation, the human feedback it receives is directly relevant to improving the current policy, not some stale earlier version. This is the same insight that makes on-policy RL algorithms like PPO more sample-efficient in many settings compared to off-policy replay-based methods.
Second, the affirmative nudge stabilizes online training, making it possible to sustain the benefits of on-policy learning without the algorithm eventually destroying itself. Without the nudge, online RLHF tanks, and the effective performance gain is much smaller because you have to checkpoint early or reduce the learning rate.
Third, information-directed exploration squeezes more value from each human comparison by choosing which responses to compare. Rather than randomly pairing two samples from the current policy, the algorithm identifies the pair that will most reduce its uncertainty about the reward function. This is where the ENN earns its keep: it provides a tractable, calibrated measure of where the reward model is uncertain, and the algorithm exploits that to make every human judgment maximally informative.
The paper includes some illustrative examples of this in action. For a sentiment classification prompt, the "infomin" pair (least informative) consists of two responses that both say "Positive" — no human judgment between those tells the reward model anything new. The "infomax" pair contrasts "positive" against "Neutral" — a genuinely ambiguous case whose resolution teaches the model something. For a reading comprehension task, the infomin pair gives two nearly identical correct answers, while the infomax pair provides two correct answers with substantially different reasoning chains, forcing the human rater to express a preference about explanation style and depth.
What About Reward-Model-Free Approaches?
An interesting negative result in the paper is that reward-model-free approaches — methods like DPO variants that directly optimize the policy from preference data without maintaining a separate reward model — were not competitive. The authors tried these and found they offered only marginal improvement over offline RLHF. This goes against the recent trend in the field where DPO and its variants have been touted as simpler alternatives to full RLHF. The paper's results suggest that maintaining an explicit, incrementally updated reward model is essential for the kind of sustained online learning they demonstrate. The reward model acts as a form of memory and generalization: it compresses what has been learned from past comparisons into a signal that can be used to evaluate new responses the human has never seen.
Caveats and Open Questions
There are several important caveats to keep in mind.
The human feedback is simulated, not real. The simulator is a Gemini 1.5 Pro-based reward model, which means the "human" preferences are actually the preferences of another (much larger) neural network. Real human preferences are noisier, more inconsistent, subject to annotator fatigue, and influenced by factors that a neural simulator might not capture. The 1,000x claim depends on the scaling law extrapolation holding up at data volumes that have not been tested, and on real human preferences behaving similarly to simulated ones.
The computational cost is also higher per label for the online and information-directed algorithms. You need to run inference on the current policy (generating 16 responses per prompt), update the reward model and policy after every batch, and for the ENN version, run inference through 100 ensemble heads to compute variance. The paper does not provide a detailed compute cost comparison, and in practice the economics depend on the relative costs of human annotation versus GPU time. Given that human annotation is often the bottleneck and the most expensive component in RLHF pipelines, a 10-1000x reduction in the number of human labels needed could easily justify significantly higher per-label compute.
The paper also explicitly states that it does not provide sufficient detail to reproduce their results. While the high-level algorithm descriptions are clear, many implementation details (learning rates, clipping thresholds, exact EMA coefficients, the value of the nudge ε, and likely many others) are unspecified. This is common in industry papers but worth noting.
Broader Implications
If these results generalize, the implications are significant along several dimensions.
For alignment economics, the cost of human feedback is a major factor in post-training budgets. If you can get the same alignment quality with 10-1000x fewer labels, the cost of RLHF drops by one to three orders of magnitude. This could make high-quality alignment accessible to organizations that cannot afford millions of human annotations.
For scaling laws research, the paper's key insight is that plotting performance on a log scale reveals qualitative differences between algorithms that linear-scale plots obscure. The authors point out that many papers in the RLHF literature use linear x-axes, which makes all algorithms look roughly similar. The log-scale perspective reveals that online methods and exploration-guided methods live on fundamentally different scaling curves. This reframing could change how the community evaluates and compares RLHF approaches.
For uncertainty estimation, the paper provides a practical demonstration that epistemic neural networks can be scaled to 9B-parameter language models with minimal overhead and can produce uncertainty estimates that are useful enough to drive meaningful exploration gains. This is an area where the gap between theory and practice has been frustratingly wide, and seeing it work at LLM scale is noteworthy.
Future Directions
The paper outlines several promising extensions. Prompt selection is one: the current algorithm selects which responses to compare given a fixed prompt, but it could be extended to also select which prompts to present, further improving data efficiency. Multiturn dialog is another natural extension — the current setup only evaluates single-turn responses, but real assistant interactions involve extended conversations where value models predicting future rewards could guide exploration. The authors also suggest extending these methods to AI agents, where actions have delayed consequences, and to AI-assisted feedback, where the model itself helps frame comparisons to make human judgments easier and more informative.
Bottom Line
This paper makes a compelling case that the RLHF community has been leaving enormous gains on the table by treating data collection as a passive, offline process. The combination of fully online learning, a simple stabilization trick (the affirmative nudge), and principled uncertainty-driven exploration produces dramatic improvements in how much the model learns from each human comparison. The demonstrated 10x gain is solid, and while the projected 1,000x gain requires trusting an extrapolation, the underlying logic — that the gap between on-policy exploration-guided learning and offline learning widens with scale — is theoretically well-motivated. For anyone working on RLHF pipelines, the message is clear: the data collection strategy matters at least as much as the optimization algorithm, and probably more.