Robotics foundation models have made remarkable progress. Today’s best systems can follow natural language instructions to pick, place, sort, and manipulate a wide variety of objects. But as these models grow more capable, evaluating them rigorously has become one of the field’s hardest unsolved problems. In this blog post, we introduce the key problems and our method for addressing them.
Why current benchmarks fall short
Real-world testing is expensive, slow, and difficult to reproduce. For a robot’s performance in the real world to be evaluated thoroughly, we need a reasonable proxy. Simulation is the natural place to run large-scale robot evaluations. Yet most existing benchmarks share a few critical issues.
Visual domain overlap in training and evaluation
First, the data and environments used in policy training and evaluation are almost always drawn from the same visual source. When a model is fine-tuned on simulated data and evaluated in that same simulated environment, strong performance reveals only that the model memorized the setup, not that it can generalize. This remains a critical issue in robot evaluations, as the visual quality of simulation hasn’t achieved parity with real-world image observations. Real2sim approaches address this issue by reconstructing photorealistic environments from real-world images using techniques like Gaussian Splatting, but per-scene setup can exceed an hour, making large-scale testing impractical.
Benchmark saturation
Second, generating tasks is a tedious endeavor. Most benchmarks have a fixed task set that is rarely updated. This quickly leads to performance saturation: models quickly max out scores on static task sets, making it impossible to distinguish which model is genuinely more capable. When every system reports over 90% success on the same benchmark, the numbers become less meaningful.
Diagnostic gap
There’s also a deeper diagnostic gap. A binary success/failure score doesn’t explain why a robot failed. Was it confused by the object’s color? Instruction phrasing? A shifted camera? Did it perform the task efficiently according to the specific language instruction? Without answers to these questions, researchers have little to act on.
Statistical trustworthiness
Every physics engine and policy is subject to some stochasticity. A single success rate on N rollouts tells you almost nothing about how confident you should be in a policy’s true performance. If a policy succeeds nine out of ten times, is it a “90% success” policy, or could it just as easily be an 80% or 95% policy that got lucky on a small sample? To investigate this, we look at the Clopper-Pearson method.
The Clopper-Pearson method is an “exact” method for constructing a binomial confidence interval around the success rate, computed directly from the binomial distribution. Let’s look at the following example: For an observed 90% success rate with just 70 rollouts, a 95% Clopper-Pearson confidence interval spans a full 15.4 percentage points (80.5% to 95.9% success rate). With 1,030 rollouts, this error tightens to a ±2 percentage-point band (88.0% to 91.8% success rate). Most published benchmarks do not run a sufficient number of rollouts to achieve statistical significance when comparing the performance of two policies.
Introducing RoboLab
We built a simulation benchmarking platform called RoboLab to address these issues. RoboLab is built around three principles:
- Enable robot-agnostic evaluations of the tasks while providing meaningful metrics
- Enable rapid generation of new tasks to avoid benchmark saturation, with support for agentic AI workflows
- Provide a full suite of analysis tools that paint a full picture of how well a policy is doing, when it fails, and why it fails.
Robot benchmarking in the age of agentic AI
RoboLab mirrors a real-world setup procedure: place objects, add a language instruction, and run a policy. Given a library of objects, users can simply place the objects in the scene, and specify a language instruction (or three!) for the task, with the whole process taking only minutes. RoboLab also comes with agent skills that can be leveraged by a coding agent to generate novel tasks directly in a user’s workflow. This efficiency also future-proofs the benchmark: new tasks can be added and outdated ones retired as generalist models improve.
Bring-your-own-robot
Building a generalist robot policy requires solving a long tail of specific tasks, and no single team has abundant data across every embodiment. A lab might have thousands of hours on a Franka arm but almost none on a humanoid, or vice versa. A benchmark tied to one specific robot forces every user into that same data gap, regardless of what they’re actually trying to build or test.
RoboLab tasks are robot- and policy-agnostic, meaning the same set of tasks can be evaluated regardless of robot embodiment or policy architecture. Users are free to make their own design choices; RoboLab simply compiles the same scenes and tasks against whichever robot they bring. This also makes sense as the number of robot embodiment choices increase in the future; it matters less which robot was used for data generation and training, only that it solved the task.
Capability-specific tasks
A useful benchmark needs to isolate distinct capabilities, not just measure whether a robot completes a task. We have observed that general-purpose manipulation draws on at least three separate competencies:
- Visual competency tests whether a policy can recognize and act on perceptual attributes like color, size, and semantic category, such as distinguishing the small red cup from other objects on the table.
- Procedural competency evaluates action-oriented reasoning: stacking objects, reorienting them, or inferring how to interact with a tool.
- Relational competency probes spatial and linguistic logic, including conjunctions (“pick the orange and the lime”), counting, and relative positions like left of or inside.
By designing tasks that each target one or more specific capabilities, we can ensure broad coverage across the full space of skills a general-purpose policy needs. In RoboLab-120, our initial benchmark of 120 human-curated tabletop pick-and-place tasks, each task is tagged with the multiple capabilities it requires, so the benchmark’s coverage across competencies remains explicit and balanced, and adjusted as new tasks are added.
| Competency | What It Tests | Example Task |
| Visual | Color, size, semantic recognition | “Put the small red cup in the bin” |
| Procedural | Stacking, reorientation, affordances | “Put all the mugs right-side-up and stack the red ones on the shelf” |
| Relational | Spatial logic, counting, conjunctions | “Pick the orange or the lime and put it in the bowl” |
Evaluating robot policies
What metrics demonstrate a robot policy is “good”?
Success rate alone tells you almost nothing about how a robot performed a task, only whether it crossed the finish line. A policy that grasps the correct object but drops it early can register as a failure, while one that succeeds only after jerky, meandering, or slow motion can register as a success. Neither case is captured using binary success.
To address this, RoboLab uses three additional evaluation tools that together paint a more complete picture of policy behavior:
- Graded task scores: Partial credit for completing subtasks within a multi-step instruction, so a robot that grasps the right object but misses the drop target isn’t scored the same as one that does nothing at all.
- Trajectory quality: Measuring motion efficiency via path length and SPARC (Spectral Arc-Length), a human-aligned metric that captures smoothness through the Fourier spectrum of velocity. Shorter, smoother motions are preferred.
- Speed of execution: Measures end effector velocity, another human-aligned metric that captures the human’s perception that faster motion is preferred.
When do robot policies fail?
Knowing how a task went wrong is just as important as knowing that it did. Beyond the usual performance metrics, RoboLab digs deeper into why a policy succeeds or fails and exactly where in the process things break down. Failure event logging automatically tracks wrong-object grasps, dropped objects, and gripper collisions, pinpointing precisely where task execution derails. Let’s observe this task: “Put all plastic bottles away in the bin” task. The policy picked up all the plastic bottles and placed it inside the bin; however, it also placed an additional orange in the bin. One could observe that technically, the task was successfully achieved! A task can technically be completed according to specification, yet the robot may still grasp the wrong object along the way before recovering.
To inspect these events, RoboLab includes a built-in dashboard that surfaces events as they happen during an episode, so users can jump straight to the frame where a failure occurred. This turns diagnosis from a manual, after-the-fact guessing game into something closer to a debugger for robot behavior: instead of asking “did it work?”, you can ask “where exactly did it stop working, and what was the context that led to that event?”
How robust is your robot policy against increasing complexity?
Real-world deployment rarely offers the clean, controlled conditions of a benchmark. Instructions come phrased in countless ways, scenes are often cluttered rather than sparse, and tasks can stretch across many steps rather than just one or two. To understand whether a policy is truly robust, we must analyze performance against increasing complexity in language, scene, and task horizon.
Language complexity
A robot that only understands precisely worded commands is of limited use outside the lab, since people naturally phrase instructions in varied and imprecise ways. Testing against multiple language instructions reveals how much a policy depends on exact phrasing versus genuine task understanding. RoboLab enables users to specify multiple language instructions in their task specification, and choose which variant to use at runtime. In our initial benchmark, we provide 3 variants: vague, default, and specific. We find that vague instructions consistently lead to failures, indicating that current models remain brittle to phrasing. We also find that sometimes having too many details in the instructions can also lead to degraded performance.
Scene complexity
Real environments are rarely as tidy as training scenes, often containing distractor objects, clutter, and visual noise that can confuse object identification. Evaluating performance as scene complexity increases shows whether a policy can still isolate the right target amid visual distractors.
Task complexity: short vs. long task horizon
Many real-world tasks aren’t single-step actions but sequences of dependent subtasks, where small failures early on can cascade into complete task failure later. For example, a task such as “Put away mugs in the cabinets” may require opening the cabinet first before grabbing the mug. Measuring how performance degrades as task horizon grows reveals how well a policy sustains accuracy over extended reasoning chains. Task designers can specify the expected sequence of subtasks in RoboLab tasks and track how well the policy progresses along. We find that most policies struggle with long-horizon tasks, with no policy able to perform more than four complex subtasks successfully.
How sensitive is your robot policy against variations?
Certain environment variations can cause performance drops, but at scale, testing each variable in isolation quickly becomes intractable. Instead, we run evaluations across many scene variations simultaneously and apply sensitivity analysis, which identifies which environmental variables are most associated with success or failure, turning intuitions like “camera placement might matter” into quantified findings.
Given episode rollouts under variation \(\theta\) with observed outcome \(x\) (for example, task success), the posterior distribution \(p(\theta \mid x) \propto p(x \mid \theta)p(\theta)\) characterizes which conditions \(\theta\) are most associated with the outcome \(x\). We estimate this posterior using Neural Posterior Estimation (NPE), which lets us pinpoint exactly which environmental variable is responsible for a given performance drop, rather than guessing at each factor’s impact one at a time.
Why it matters
Robotics benchmarking still lags far behind the rest of AI research, and without a field-standard benchmarking platform, it is difficult to measure progress. As policies grow more capable, success rates alone won’t tell us whether a model truly generalizes or just memorized its test conditions, and that gap will only widen as models improve. The path forward requires evaluation that evolves as fast as the models it measures: benchmarks that expand rather than saturate, metrics that diagnose rather than simply score, and analysis that tells researchers not just how well a policy performs, but how to improve it. RoboLab establishes a scalable path toward diagnostic robot evaluation for real-world policies using simulation.
For more information about RoboLab, check out the paper and code on GitHub. RoboLab was developed by NVIDIA Research, including researchers with affiliations at the University of Sydney and University of Toronto.
RoboLab research powers NVIDIA Isaac Lab-Arena, an open source simulation framework for large-scale policy setup and evaluation. Key RoboLab features are planned for productization in August 2026.
- Website: https://research.nvidia.com/labs/srl/projects/robolab
- Paper: https://arxiv.org/abs/2604.09860
- Code: https://github.com/NVLabs/RoboLab
Acknowledgements
The author thanks Alex Zook, Alperen Degirmenci, Ankit Goyal, Elie Aljalbout, Fabio Ramos, Hugo Hadfield, Jonathan Tremblay, Karl Pertsch, Moritz Reuss, Rishit Dagli, Stan Birchfield (alphabetically listed) for insightful discussions throughout our work on large-scale robot evaluations.
Citations
@misc{yang2026benchmarking,
title = {How to evaluate real-world policies for general-purpose robots},
author = {Yang, Xuning},
year = {2026},
month = {July},
organization = {Seattle Robotics Lab (SRL), NVIDIA},
howpublished = {https://developer.nvidia.com/blog/how-to-evaluate-general-purpose-robot-policies-for-real-world-deployment},
note = {Blog post},
}References
Yu, T., et al. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. CoRL 2019. https://arxiv.org/abs/1910.10897
Liu, B., et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023. https://arxiv.org/abs/2306.03310
Zhu, Y., et al. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv 2020. https://arxiv.org/abs/2009.12293
Mu, Y., et al. RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins. CVPR 2025. https://arxiv.org/abs/2504.13059
Jain, A., et al. PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies. arXiv 2025. https://arxiv.org/abs/2512.16881
Jangir, Y., et al. RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation. arXiv 2025. https://arxiv.org/abs/2510.23571
Li, X., et al. Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024. https://arxiv.org/abs/2405.05941
TRI LBM Team et al., “A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation”, Science Robotics, 2026, https://arxiv.org/abs/2507.05331
Frazier, D. T., et al. “The Statistical Accuracy of Neural Posterior and Likelihood Estimation.” 2024, https://arxiv.org/abs/2411.12068
Black, K., Brown, N., Driess, D., et al. “\(\pi\) 0: A Vision-Language-Action Flow Model for General Robot Control.” 2024, https://arxiv.org/abs/2410.24164
Pertsch, K., Stachowicz, K., Ichter, B., et al. “FAST: Efficient Action Tokenization for Vision-Language-Action Models.” 2025, https://arxiv.org/abs/2501.09747
Black, K., Brown, N., Driess, D., Esmail, A., et al. “\(\pi\) 0.5: a Vision-Language-Action Model with Open-World Generalization.” CoRL 2025, https://arxiv.org/abs/2504.16054
Beyer, L., Steiner, A., Pinto, A. S., et al. “PaliGemma: A versatile 3B VLM for transfer.” 2024, https://arxiv.org/abs/2407.07726
