The Right Kind of WrongCounterfactual Data for Reward Learning in Manipulation

TL;DR Reinforcement learning (RL) can improve robot policies beyond imitation learning, but it requires rewards that capture task progress and failure. Learning such rewards requires not only successful demonstrations, but also informative counterfactual failures that can guide the model on behaviors to penalize. However, large-scale robot datasets are dominated by successful behavior, and collecting failures on real hardware is expensive. This raises a practical question: given a limited data-collection budget, which sources of counterfactual experience are worth investing in? We compare four sources of failure data: kinesthetic teaching (KT), policy rollouts, human videos, and real-to-sim rollouts, across five real-world manipulation tasks. By keeping the task suite, successful demonstrations, and the evaluation setup fixed, we study the effect of these sources through LoRA adaptation of a reward model and few-shot prompting of a frozen model in two different data scale scenarios. We evaluate the resulting reward models on a custom held-out benchmark of 250 human-annotated policy rollouts, measuring trajectory ranking, partial-completion calibration, and success detection. Finally, we use selected reward models to guide RL post-training of π₀ and test whether improvements in offline reward quality translate into better robot behavior. In sum, our study examines whether inexpensive counterfactual sources can match or surpass smaller amounts of faithful real-robot failure data when scaled, for improving rewards models in robot manipulation.

Four sources of failure

Each source records the same five tasks. Every failure is graded by how far it got; here are low, medium and high progress failures on Pour for each source.

RL results

Each reward model post-trains π₀ with DSRL on the real robot, evaluated over 20 trials per task. Pick a task to see one rollout per reward model.

Evaluation videos play at 20× speed. Each clip is labelled with its reward source and success rate.

Reward curves

Predicted progress from the reward model adapted on each source, on held-out rollouts from our 250-clip benchmark. Pick a task and a clip, then play or scrub. The dotted line is the human-graded credit for that clip.

BibTeX

@article{thiruvenkadam2026counterfactual,
  title   = {The Right Kind of Wrong: Counterfactual Data for Reward Learning in Manipulation},
  author  = {Thiruvenkadam, Gokul Kannan and Patel, Vishal M. and Bharadhwaj, Homanga},
  year    = {2026}
}