Developing autonomous vehicle (AV) policies requires bridging an important gap between training and deployment. Vision-language-action (VLA) models that can reason over more complex driving scenes and produce richer intermediate reasoning are predominantly trained in open-loop, where model outputs are directly compared to ground-truth behaviors without considering their effect on the environment.
In deployment, however, a driving policy runs in closed-loop, where every braking, steering, and navigation decision affects the environment, and small errors can compound over time.
A systematic means to address this challenge is provided by NVIDIA Alpamayo, an open portfolio of AI models, simulation frameworks, and physical AI datasets for AV development. Alpamayo includes the AlpaSim AV simulation platform and the AlpaGym closed-loop training framework.
This post explains how to train AV models in closed-loop with NVIDIA Alpamayo. Specifically, it walks through how to:
- Install and configure AlpaGym
- Define closed-loop rewards
- Launch closed-loop training
- Export the post-trained checkpoint for downstream use
Closed-loop post-training with AlpaGym extends AV training workflows by turning AlpaSim rollouts into training experience. Rather than treating simulation only as a final evaluation stage, AlpaGym connects simulator feedback directly to the policy training loop.
How to use AlpaGym for closed-loop reinforcement learning
Reinforcement learning (RL) can be used to improve a policy that was initially trained in open-loop. Instead of optimizing only against logged expert trajectories, the model can now learn from the consequences of its own actions in simulation.
This shift is critical for AV development, where small prediction or planning errors can compound over time. In closed-loop training, each braking, steering, and navigation decision affects the next state of the environment, revealing failure modes that static datasets or open-loop evaluation may miss.
However, enabling closed-loop RL comes with its own challenges. Model inference, running simulation, training models, syncing weight updates, communicating across instances and moving data—all in parallel—is complex. This requires orchestration and efficient utilization of compute resources in a robust yet flexible manner.
To address these challenges, AlpaGym connects policy training to AlpaSim closed-loop rollouts and provides an open source, high-throughput framework for closed-loop RL. The system combines AlpaSim simulator microservices, NVIDIA Physical AI Open Datasets, and distributed NVIDIA Cosmos-RL training framework into a scalable post-training pipeline.
Built to scale seamlessly from a single GPU to multi-node GPU clusters, AlpaGym supports efficient large-scale training through an asynchronous and stable distributed RL pipeline, without requiring changes to user code. It integrates AlpaSim and Cosmos RL as its runtime and orchestration layer, GRPO as a default algorithm, and includes reference reward functions tested with Alpamayo models and the Physical AI AV NuRec dataset.
To get started with AlpaGym post-training, follow the steps outlined below.
Step 1: Install and configure AlpaGym
Recommended: a CUDA GPU with >=40 GB VRAM and ~100-150 GB free disk space (env + container images + ~21 GB weights), plus ~1.5 GB per NuRec scene (~1.5 TB for the full public_2601 suite).
To install AlpaGym from the Alpamayo checkout, install the native CUDA dependencies and Redis on the host, then sync the UV workspace:
# uv, plus Docker Compose (AlpaSim Wizard runs on Docker Compose).
uv self update
docker compose version
# CUDA apt keyring repo (required before the cuDNN package resolves).
wget -O /tmp/cuda-keyring_1.1-1_all.deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i /tmp/cuda-keyring_1.1-1_all.deb
sudo apt-get update
# Host dependencies: cuDNN + NCCL headers, Redis, Git LFS.
sudo apt-get install -y libcudnn9-dev-cuda-12 \
libnccl-dev=2.26.2-1+cuda12.8 libnccl2=2.26.2-1+cuda12.8 \
redis-server git-lfs
git lfs install
git lfs pull
# Request access to the gated NuRec dataset, then authenticate.
hf auth login
# Or export HF_TOKEN=...
# Optional: Weights & Biases logging (add cosmos.logging.logger='[console,wandb]' to the run).
export WANDB_API_KEY=...
# Confirm host tools/headers, then sync the workspace.
install/check_env.sh
uv sync --all-packages
The Python environment is managed by uv, but cuDNN, NCCL, and the redis-server binary are host dependencies used by the CUDA model stack and Cosmos-RL. Alternatively, a suitable Dockerfile is also provided. Hugging Face authentication is required to download the scene artifacts.
An AlpaGym run is a Hydra configuration. It specifies the policy checkpoint, the AlpaSim scene set, rollout parallelism, reward function, and Cosmos-RL training parameters. In this workflow, the starting checkpoint is an Alpamayo model.
Step 2: Define the closed-loop reward
The reward should match the behavior you want to improve in closed-loop. For trajectory-quality post-training, common reward terms include progress, lane keeping, collision avoidance, offroad rate, comfort, and distance to a reference trajectory.
A practical first reward is intentionally simple: combine progress with penalties for safety-critical failures. In AlpaGym, this can be expressed as a small sum of terms, using AlpaSim metrics where possible:
# reward/progress_safety.yaml
terms:
- kind: metric
metric_name: progress
scale: 1.0
- kind: metric
metric_name: collision_any
scale: -10.0
- kind: metric
metric_name: offroad
scale: -5.0
- kind: distance_to_gt
scale: -0.01
Once the pipeline is stable, add more targeted terms for the failure modes observed in AlpaSim videos and metrics.
Step 3: Launch closed-loop post-training
Start AlpaGym training from your model checkpoint. AlpaGym currently supports the Alpamayo 1.5 model (10B parameters), which serves as an example model here, with further detailed instructions in the NVlabs/alpagym GitHub repo for how to use your own AV model.
First, download the Alpamayo 1.5 checkpoint and convert it into an AlpaGym-compatible checkpoint:
uv run --no-sync python -c "\
from huggingface_hub import snapshot_download; \
print(snapshot_download('nvidia/Alpamayo-1.5-10B', \
local_dir='./tmp/checkpoints/Alpamayo-1.5-10B'))\
"
uv run --no-sync --package alpagym-runtime python \
packages/policies/alpamayo_r1/scripts/convert_release_to_alpagym_checkpoint.py \
--input ./tmp/checkpoints/Alpamayo-1.5-10B \
--output ./tmp/checkpoints/alpamayo-1.5-10B_alpagym_ckpt \
--overwrite
uv run --no-sync --all-packages python -m alpagym_host.cli \
experiment=alpamayo_1_5_local_2gpu_smoke \
policy.model.path="$(pwd)/tmp/checkpoints/alpamayo-1.5-10B_alpagym_ckpt" \
reward=progress_safety
This will bring up AlpaGym with AlpaSim on two GPUs (tested on 2x 50GB RTX 6000 Ada); note that training the 10B model does not fit on one GPU. A distillation script to a smaller single-GPU-compatible checkpoint is planned. To see examples and instructions on how to scale to multiple GPUs, including single- or multi-node deployments, visit the NVlabs/alpagym GitHub repo.
During training, AlpaGym requests scene rollouts from AlpaSim, collects per-episode artifacts, computes rewards, and updates the policy. Useful training signals include mean reward, reward variance, failure rates, policy loss, rollout throughput, and the gap between generated rollouts and the latest policy weights.
In this recipe, these rollout artifacts and training signals are the primary outputs of the post-training run. They help you confirm that closed-loop learning is running correctly and select checkpoints for downstream evaluation on your own held-out AlpaSim scenario suites. Run artifacts are written under tmp/alpagym-runs/ and Hydra output under outputs/.
Step 4: Export the post-trained checkpoint
After training, the AlpaGym run writes a Cosmos-RL safetensors export under tmp/alpagym-runs/. The AlpaSim driver cannot load that training export directly, so first convert it to inference format (the inverse of the checkpoint conversion in Step 3):
# 1. Convert the training export to inference format (the driver cannot load it raw).
# Each run writes an export at tmp/alpagym-runs/<run>/cosmos/<timestamp>/safetensors/step_<N>/.
uv run --no-sync --package alpagym-alpamayo-r1 python \
packages/policies/alpamayo_r1/scripts/convert_alpagym_checkpoint_to_inference.py \
--input tmp/alpagym-runs/<run>/cosmos/<timestamp>/safetensors/step_<N> \
--output ./tmp/checkpoints/alpamayo-1.5-10B_clrl_inference --overwrite
Next, run the converted checkpoint on a representative scenario to verify that the policy, driver, and simulation loop are connected correctly. At this stage, you can inspect how the policy behaves when its own actions affect the next state of the environment. Eval needs access to the gated nvidia/Cosmos-Reason2-8B VLM, which the driver uses to rebuild the base tokenizer/preprocessor.
# 2. Run eval from the AlpaSim checkout AlpaGym already cached.
cd "$(ls -d ~/.cache/alpagym/alpasim/*/ | head -n1)"
uv run alpasim_wizard \
deploy=local topology=1gpu \
driver=alpamayo1_5 \
defines.drivers=/ABS/PATH/TO/alpagym_public/tmp/checkpoints \
driver.model.checkpoint_path=/mnt/drivers/alpamayo-1.5-10B_clrl_inference \
scenes.scene_ids=[clipgt-9ea70552-6dcb-4ee8-a368-9a906a333f6e]
A closed-loop rollout provides useful qualitative signals: whether the model produces stable trajectories and remains within the drivable area, how it reacts to nearby traffic agents, and which failure modes should be targeted during post-training.
With this checkpoint, teams can inspect rollout videos, per-episode metrics, reward traces, and failure cases collected during training. These artifacts are useful for debugging reward design, checking rollout stability, and selecting checkpoints for later held-out evaluation in AlpaSim.
Get started post-training AV models
Closed-loop post-training provides a practical path for iterating on end-to-end driving policies. In this case, AlpaGym uses closed-loop rollouts to post-train AV policies in simulation, enabling them to learn from the consequences of their actions.
You can use these tools together with the other components of the NVIDIA Alpamayo Open Platform to develop reasoning models that can be run, inspected, and post-trained in a closed-loop simulation workflow. Extend this same recipe more broadly with your own rewards, scenarios, and evaluation suites.
Ready to get started? Explore the broader AlpaGym framework and check out the NVlabs/alpamayo-recipes GitHub repo to adapt the recipe in this post for your own use cases.
To evaluate your model on a public leaderboard, see the two open AV challenges NVIDIA launched at CVPR 2026:
To learn more, see these related resources:
- Expanding the Alpamayo Open Platform for Developing Reasoning AVs Across Models, Data, and Simulation
- Taking Alpamayo to New Heights with Driving Foundation Models and Closed-Loop Training
Join NVIDIA founder and CEO Jensen Huang for the NVIDIA GTC Taipei 2026 Keynote and dive deeper with related sessions.
