Agent Substrate: Advancing Kubernetes for Agents at Scale

AI agents spend most of their time idle, waiting on a human prompt, a tool result or a model response. Kubernetes wasn’t built for that. Running one Pod per session holds memory for sessions that aren’t doing anything, and deleting Pods between turns pays a multi-second cold start every time and loses the agent’s state. RL post-training hits the same wall.

Agent Substrate, now donated to the Cloud Native Computing Foundation (CNCF) Sandbox, is an open-source, security-hardened, zero-trust runtime for sandboxed agents, engineered to safely run millions of agents and tools with 10x higher density than standard containers. It multiplexes many sessions over a small pool of worker Pods. Idle sessions are checkpointed, memory and filesystem included, and restored onto whichever worker is free when the next request arrives. As agentic infrastructure fragments into proprietary runtimes that bypass Kubernetes, NVIDIA is contributing upstream to keep this work open.

This post covers which workloads suit Agent Substrate, what is being built upstream, and how the project works with KAI Scheduler and NVIDIA Dynamo components in the DSX AI Factory Platform.

The shape of an agent workload

Traditional web services run as a relatively small number of long-lived instances. Agents follow a different pattern: they wake up and take one step, running a shell command, editing a file, calling a test suite or calling an LLM. Then they wait, sometimes for milliseconds and sometimes for minutes, before the next step. The agent’s state (its filesystem, its processes, its working directory) has to continue between steps.

Kubernetes struggles because spinning up a compute Pod for an agent means a trip through the API server, admission, etcd, the scheduler, the kubelet, image pulls and network setup. That takes seconds, which is fine for a service that runs for weeks. It’s far too slow for a sandbox that must appear, run a 10 ms bash command and get out of the way, hundreds of times per second.

This leaves platform teams with two suboptimal choices: either keep one Pod warm per agent and pay for compute that is idle almost all the time, or create Pods on demand and pay a cold start penalty on every step.

How Agent Substrate works

Agent Substrate uses Kubernetes for running a pool of ready pods. It creates a separate control plane that maps a large set of actors (agents and their environments) onto that smaller pool of workers. When an actor goes idle, it is snapshotted out of its sandbox (gVisor or a microVM) and its worker is freed. When the actor is called again, it is restored into whichever worker is free, with its memory and filesystem state intact.

Waking an actor doesn’t require a new trip through the API server, scheduler or kubelet. Agent Substrate restores the actor onto an existing worker instead. The worker pods are still ordinary Kubernetes Pods, so cluster scheduling, quotas and placement policies still apply as we describe later.

The use cases driving demand

We see three kinds of workload driving demand for Agent Substrate:

• Coding assistants and agent serving: A coding agent alternates between LLM calls and work in its own environment: cloning a repository, editing files, running builds and tests. Users expect the environment to persist between turns and to respond immediately. A platform serving thousands of these sessions can’t afford a dedicated warm sandbox for each one.
• Reinforcement Learning (RL) post-training with agentic environments: RL with verifiable rewards runs GPU trainers and generation servers alongside a fleet of CPU-heavy environments (code sandboxes, compilers, browsers, simulators) that execute actions and check results. Sandbox demand arrives in bursts as rollout batches dispatch actions. A training run can grow from a few hundred sandboxes in early experiments to many hundreds of thousands in production. The slowest environment determines when the trainer gets its next batch, so cold starts on that path leave GPUs idle.
• GPU sandboxes for research: A researcher’s coding agent needs a GPU, not to serve a model but to compile CUDA code, benchmark a kernel or run a numerical calculation. It launches a GPU sandbox on a shared cluster, runs the job and deletes the sandbox. Supporting this across an organization requires many short-lived, isolated GPU sandboxes that start quickly and share a limited pool of GPUs fairly.

Upstream work: What’s landed and what’s next

Each contribution below started from one of those use cases. All of this work is public in the Agent Substrate repository:

• When every worker is busy, the next request waits instead of failing: In an oversubscribed system, such as a coding-assistant platform at peak or an RL rollout burst, a user’s next turn can arrive while every worker is busy. Agent Substrate used to return an error immediately, even though another actor would usually free a worker a moment later.

Request parking in the Agent Substrate router (#221) holds the request and retries the resume until a worker frees up, with a bounded wait and a cap on how many requests can wait at once. A follow-up (#1229) makes sure requests to already-running actors never get stuck in that line.

• Giving agents a GPU inside their sandbox: The research GPU sandbox described above, along with GPU-backed RL environments such as simulators.

We are working on device-pluggability that adds device support on Kubernetes Dynamic Resource Allocation (DRA). Device vendors plug in their devices through a profile, each container in a sandbox requests only the devices it needs (#752), and the same model works for both gVisor and microVM sandboxes. NVIDIA GPUs will be the first supported device.

• One container can’t starve or kill its neighbors: An agent sandbox often runs the agent’s workload next to helper containers for logging, proxying or metrics. Some of those may be untrusted. A runaway container shouldn’t consume the whole sandbox’s memory and take the agent down with it.

Per-container CPU and memory limits for microVM sandboxes (#859) addresses that issue. A container that exceeds its memory limit is stopped on its own, and the rest of the sandbox keeps running.

Preserving GPU sandboxes across suspend and resume

When an agent pauses to wait for human review, its sandbox should release the GPU and resume later with its compiled artifacts and loaded data still in place. Supporting that requires saving and restoring both the agent’s state and the GPU state.

Restoring GPU state introduces hardware compatibility constraints. A GPU checkpoint can usually be restored only onto the same GPU model, and sometimes only with the same driver version. Agent Substrate proposes a design to address this.

Device vendors describe how to pause their devices safely and which device properties a snapshot depends on. Agent Substrate records those properties with each snapshot and restores the sandbox on a matching device, or falls back to a clean cold start if none is available. The design is vendor-neutral and builds on DRA. NVIDIA GPUs are the first target; the design can be extended to other hardware types.

NVIDIA Dynamo: Connecting the agent loop to the inference loop

Agent Substrate manages where agents act; inference runtimes manage where they think. Both face a similar efficiency challenge: keeping idle sandboxes warm wastes CPU and memory, while rebuilding model context on repeated model calls wastes GPU cycles. Scaling each layer independently improves utilization. Coordinating them can also reduce latency and preserve reusable context as agents move between tool execution and model calls.

NVIDIA Dynamo, the open-source distributed inference framework, provides that coordination. Agent Substrate activates, suspends and restores stateful agent environments across a pool of warm sandbox workers. Dynamo orchestrates inference across GPU workers. The Dynamo router sits between them and connects what the agent is doing with what is happening on the GPUs.

Making inference agent-aware

Traditional inference routing treats every model call as an independent request. In an agent, each request is one step in a longer loop, and consecutive requests often reuse the same system prompt, tool definitions, conversation history and earlier observations.

Dynamo’s router sees where reusable key-value (KV) cache lives, which workers are busy with prefill or decode, and how load is spread across the GPU fleet. Agent Substrate and the agent harness add signals such as session identity, request priority, expected output length, tool-call boundaries and lifecycle events.

With both, Dynamo can:

• Keep related model calls on workers that already hold reusable KV cache, without creating overloaded GPU hot spots
• Prioritize interactive turns over background agent work when the inference pool is under pressure
• Keep system prompts, tool definitions and conversation history available across tool-call gaps
• Drop cache state from terminated subagents, closed reasoning branches or summarized contexts first
• Prepare likely context as a tool call approaches completion

Feeding GPU intelligence back to Agent Substrate

The most important part of the integration is that Dynamo exposes inference signals to Agent Substrate, helping it decide when to keep agents running, suspend them or resume them.

Dynamo is closer to GPU execution than the agent runtime. It has visibility into cache readiness, queue pressure, active prefill and decode work, and how long an LLM request is likely to take. That information can inform both inference routing and how Agent Substrate schedules agent sandboxes.

Dynamo can return signals such as projected time to first token, projected completion time, request progress, cache readiness, and GPU-pool saturation. Agent Substrate can use this feedback to decide whether to keep an Actor resident, suspend it during a longer inference wait, or resume it as the model response approaches completion.

At fleet scale, the same feedback can help tune warm-worker capacity, regulate agent and subagent fan-out, and avoid activating CPU sandboxes that would otherwise sit idle behind a saturated GPU queue.

This integration establishes a simple division of responsibility. Agent Substrate manages the high-frequency lifecycle of agent environments. Dynamo manages the high-performance lifecycle of inference. Dynamo connects the two around the loop that defines agentic computing: model call, tool call, observation, and model call again.

Get started: A coding assistant with Dynamo and Agent Substrate

This guide provides step-by-step instructions for deploying a coding assistant on Dynamo and Agent Substrate.

You’ll learn how to:

Guide link placeholder – add when supplied

• Run Pi coding agents as Substrate Actors: giving each agent an isolated, stateful environment to inspect repositories, edit code, and run tests.
• Connect the agents to GLM-5.3-Flash served through NVIDIA Dynamo: completing the loop between model inference and tool execution in Substrate-managed sandboxes.
• Use Harbor to orchestrate evaluations on SWE-bench Verified: running agents against software issues and collecting results to assess whether their changes resolve the tasks.

Agent Substrate is still evolving. Its APIs are still changing, GPU support is being rebuilt on the new DRA foundation, and parts of the Dynamo agent integration are experimental. Now is a good time to contribute and help shape the core design.

Running Agent Substrate with KAI Scheduler and Dynamo

Agent Substrate runs above Kubernetes and works alongside the NVIDIA software that serves models and places workloads. Substrate manages stateful agent environments, KAI Scheduler places the Kubernetes workloads, and Dynamo serves the models and exchanges inference-state signals with the agent runtime.

Together, they coordinate CPU-side tool execution with GPU-side inference so shared cluster capacity is used more productively.

Table 1, below, shows where each piece sits. Dynamo, KAI Scheduler and the Kubernetes foundation components are part of NVIDIA DSX OS.

Agent Substrate is an independent, open-source CNCF Sandbox project and not a named DSX OS component, though NVIDIA is contributing upstream and developing interoperability with DSX OS components.

Table 1. Where Agent Substrate fits in the stack for running AI agents and RL environments on Kubernetes: each layer’s technology, role, and relationship to NVIDIA DSX OS.

KAI Scheduler and Agent Substrate operate at different scheduling levels: Substrate worker Pods are ordinary Kubernetes Pods. KAI Scheduler can place them today alongside the GPU trainers and inference servers they support. Coordinating worker-pool autoscaling with KAI’s queues and fair-share policies could extend resource sharing across tenants to the environment layer.

Build agent infrastructure with us at KubeCon

• Agent Substrate Community Party—Tuesday, Nov. 10, Salt Lake City. An informal evening for maintainers, contributors, and anyone in the ecosystem to connect. Join us to talk about all the contributions we mentioned in this blog and meet with the people building the future of agentic infrastructure on Kubernetes.
• Lightning session—Wednesday, Nov. 11, 1-1:15 pm at Google Cloud Booth #122: “Talk From Pods to Actors : GPU-Aware Infrastructure for Agentic Workloads”
• NVIDIA booth #170 at KubeCon + CloudNativeCon North America—Nov.10–12. Come talk to our experts about agent platforms, RL post-training on Kubernetes, Substrate, Dynamo and KAI Scheduler.
• Meet with Experts at NVIDIA booth #170—Wednesday, Nov. 11, from 3:15–3:35 p.m. MT to discuss Agent Substrate and running agent workloads on Kubernetes.
• Explore the full Meet the Experts lineup—NVIDIA at KubeCon + CloudNativeCon North America 2026

Going further

If you’re building agent platforms, coding assistants or RL environments on Kubernetes, we want to hear about your workload. Find us at the event, check out the Dynamo guide or open an issue agent-substrate/substrate or ai-dynamo/dynamo. Join the community meetings and get involved in what comes next.