Running a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared across many teams, the coordination costs increase as the number of teams grows. Challenges include conflicting CRD versions, overlapping RBAC, and no clean way to carve GPU capacity into team-level budgets. At a certain scale, teams might start asking for their own clusters just to regain autonomy.
This post provides a pattern that preserves team autonomy without splitting the hardware. This solution involves a single control plane cluster with a GPU pool, GPU sharing with per-team quotas, and isolated Kubernetes control plane per team including an API server, controller, data store, syncer, and scheduler. This can be achieved using two open source tools: KAI Scheduler and vCluster.
Follow along with the tutorial steps and you’ll have three teams running real GPU Kubernetes pods in their own tenant clusters, all sharing a single physical GPU. You’ll also be able to verify that each team only sees their own workloads.
Tutorial prerequisites and notes
To keep the process reproducible for users that have limited resources, this tutorial uses a cluster with one NVIDIA L40S GPU and three teams sharing fractions of it. This makes the moving parts easy to see and try. The process works the same way on a larger cluster with hundreds of GPU nodes and dozens of teams. You can scale the node pool, the queue hierarchy, and the number of tenant clusters.
KAI Scheduler is a robust, efficient, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU resource allocation for AI workloads. It’s designed to manage large-scale GPU clusters, including thousands of nodes, and a high throughput of workloads. With KAI Scheduler, you can dynamically allocate GPU resources to workloads. It can run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is handled by KAI Scheduler. Everything else goes through the usual kube-scheduler process.
The vCluster Kubernetes platform provisions fully isolated tenant clusters on your infrastructure or directly on bare metal. Each tenant cluster gets its own API server, custom resource definitions (CRDs), and role-based access control (RBAC), indistinguishable from a dedicated Kubernetes cluster, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants: no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a natural fit for GPU infrastructure where teams need their own clean cluster experience without splitting the hardware.
The vCluster shared-nodes model is used for this tutorial, so teams share the GPU node while each gets its own isolated control plane, the right fit for trusted internal teams. For untrusted tenants needing node-, network-, and storage-level separation, the same pattern extends to vCluster private nodes.
The example in this post uses three teams: NLP Team, Vision Team, and Recommender System Team. The NLP Team wants to install their own CRDs. The Vision Team wants cluster-admin to debug scheduling. The Recommender Team is on a different Kubeflow version. Nobody wants to share a kubectl context and accidentally break one another’s environments.
Using vCluster, each team gets their own isolated control plane, RBAC, namespaces, and CRDs. They can each have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs.
Demo environment
This demo runs on an NVIDIA Brev GPU instance on Nebius with:
- One NVIDIA L40S, 40 vCPUs, 160 GiB RAM, 256 GiB disk (48 GB VRAM)
- Ubuntu 24.04.4 LTS
- MicroK8s v1.36.2 – Kubernetes was preconfigured by Brev, including the MicroK8s gpu addon, which pre-installs the NVIDIA GPU Operator into the gpu-operator-resources namespace
- KAI Scheduler v0.16.4
- vCluster CLI 0.35.1
Note: For different setups, the cluster-creation and GPU Operator install steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is identical on any Kubernetes that has the NVIDIA GPU Operator running with Container Device Interface (CDI) enabled.
Step 1: Local tooling
Install standalone kubectl and helm to avoid prefixing every command with microk8s. Then wire up a kubeconfig and pin MicroK8s so snap doesn’t auto-upgrade the control plane during the demo.
sudo snap refresh --hold microk8s
sudo snap install kubectl --classic --channel=1.35/stable
sudo snap install helm --classic
mkdir -p ~/.kube
sudo microk8s config > ~/.kube/config
sudo chown $USER:$USER ~/.kube/config
chmod 600 ~/.kube/config
Next, make sure the required MicroK8s addons are enabled:
microk8s enable dns
microk8s enable hostpath-storage # vCluster needs PVCs
Then verify:
kubectl get nodes -o wide
kubectl get storageclass
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
brev-8dq0cch1j Ready <none> 57d v1.36.2 10.0.0.20 <none> Ubuntu 24.04.4 LTS 6.11.0-1016-nvidia (amd64) containerd://2.2.3
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
microk8s-hostpath (default) microk8s.io/hostpath Delete WaitForFirstConsumer false 20m
Step 2: Add the Helm repo
Add the NVIDIA Helm repo:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
Step 3: Confirm the GPU Operator
kubectl get pods -n gpu-operator-resources
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-wpcjm 1/1 Running 0 8m
gpu-operator-57d75775c8-npjzz 1/1 Running 0 9m
gpu-operator-node-feature-discovery-... 1/1 Running 0 9m
nvidia-container-toolkit-daemonset-ft4hb 1/1 Running 0 8m
nvidia-cuda-validator-8vqdv 0/1 Completed 0 8m
nvidia-device-plugin-daemonset-l6fj6 1/1 Running 0 8m
nvidia-operator-validator-cdnmj 1/1 Running 0 8m
If you’re using an older GPU Operator, upgrade in place to 26.3.x:
helm upgrade gpu-operator nvidia/gpu-operator \
-n gpu-operator-resources \
--version v26.3.3 \
--reset-then-reuse-values
If needed, install NVIDIA GPU Operator directly:
helm install gpu-operator nvidia/gpu-operator \
-n gpu-operator-resources --create-namespace \
--version v26.3.3 \
--set driver.enabled=false \
--set operator.defaultRuntime=containerd \
--set toolkit.env[0].name=CONTAINERD_CONFIG \
--set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd.toml \
--set toolkit.env[1].name=CONTAINERD_SOCKET \
--set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
--set-string toolkit.env[2].name=CONTAINERD_SET_AS_DEFAULT \
--set-string toolkit.env[2].value=1
Step 4: Install KAI Scheduler
Next, install the KAI Scheduler:
helm upgrade -i kai-scheduler \
oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
-n kai-scheduler --create-namespace \
--version v0.16.4 \
--set "global.gpuSharing=true"
Then verify:
kubectl get pods -n kai-scheduler
NAME READY STATUS RESTARTS AGE
admission-57556f949-sp98t 1/1 Running 0 47s
binder-66785d8dd9-9frgk 1/1 Running 0 46s
kai-operator-6fdf595c4d-292d7 1/1 Running 0 52s
kai-scheduler-default-6bb667b767-vq2q4 1/1 Running 0 46s
pod-grouper-84dfc7759b-v5qtb 1/1 Running 0 47s
podgroup-controller-5878f48dbb-v7wcn 1/1 Running 0 47s
queue-controller-7796bb8984-hdn5r 1/1 Running 0 46s
Step 5: Define team queues
KAI Scheduler uses a Queue CRD to model an org → team hierarchy. This step involves creating one parent (ml-org, with a total budget of one GPU) and three child queues. Each is guaranteed 0.33 of the GPU and allowed to increase to the full GPU when others are idle.
Save the following as create-queues.yaml:
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: ml-org
spec:
resources:
gpu: { quota: 1, limit: -1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-nlp
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-vision
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
name: team-recommender
spec:
parentQueue: ml-org
priority: 100
resources:
gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 }
Here, quota is the guaranteed minimum, limit is the maximum allowed, and overQuotaWeight controls how surplus is split.
Next, apply and list:
kubectl apply -f create-queues.yaml
kubectl get queues
queue.scheduling.run.ai/ml-org created
queue.scheduling.run.ai/team-nlp created
queue.scheduling.run.ai/team-vision created
queue.scheduling.run.ai/team-recommender created
NAME PRIORITY PARENT CHILDREN DISPLAYNAME
default-parent-queue ["default-queue"]
default-queue default-parent-queue
ml-org ["team-nlp","team-vision","team-recommender"]
team-nlp 100 ml-org
team-recommender 100 ml-org
team-vision 100 ml-org
Note that default-parent-queue and default-queue are created automatically by KAI Scheduler on first install. They are the fallback queue for any pod that doesn’t specify one.
Step 6: Spin up a vCluster per team
Next, install the vCluster CLI:
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
sudo install -m 755 vcluster /usr/local/bin/vcluster
rm vcluster
vcluster --version
vcluster version 0.35.1
Define the vCluster config. The critical setting is setOwner: false. The KAI Scheduler pod-grouper walks ownership chains (Job → Pod, Deployment → ReplicaSet → Pod) to auto-group workloads. Disabling vCluster owner rewriting allows KAI Scheduler to see the real hierarchy.
cat > vcluster.yaml <<'EOF'
experimental:
syncSettings:
setOwner: false
sync:
fromHost:
nodes:
enabled: true
selector:
all: true
EOF
Then create one vCluster per team:
vcluster create team-nlp --values vcluster.yaml --connect=false
vcluster create team-vision --values vcluster.yaml --connect=false
vcluster create team-recommender --values vcluster.yaml --connect=false
Verify that all three vClusters are running:
vcluster list
NAME | NAMESPACE | STATUS | VERSION | CONNECTED | AGE
-------------------+---------------------------+---------+---------+-----------+-------
team-nlp | vcluster-team-nlp | Running | 0.35.1 | | 102s
team-recommender | vcluster-team-recommender | Running | 0.35.1 | | 88s
team-vision | vcluster-team-vision | Running | 0.35.1 | | 93s
Each team sees the real nodes, including the GPU node:
vcluster connect team-nlp -- kubectl get nodes
19:09:22 done vCluster is up and running
NAME STATUS ROLES AGE VERSION
brev-8dq0cch1j Ready <none> 6m20s v1.36.2
Step 7: Deploy a workload from each team
Each team deploys their GPU workload from their own vCluster. The pod spec is simple, with three fields telling KAI Scheduler what to do:
vcluster connect team-nlp -- kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: nlp-sentiment-model
labels:
kai.scheduler/queue: team-nlp # Which team
annotations:
gpu-fraction: "0.33" # How much GPU
spec:
schedulerName: kai-scheduler. # Use KAI, not default
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: nlp-inference
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ["bash", "-c", "nvidia-smi; sleep infinity"]
nodeSelector:
nvidia.com/gpu.present: "true"
EOF
19:13:34 done vCluster is up and running
pod/nlp-sentiment-model created
Do the same to deploy from each team, changing the queue name.
Now, verify:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
vcluster-team-nlp nlp-sentiment-model-x-default-x-team-nlp 1/1 Running 0 110s 10.1.171.180 brev-8dq0cch1j
vcluster-team-recommender recommender-model-x-default-x-team-recommender 1/1 Running 0 6s 10.1.171.187 brev-8dq0cch1j
vcluster-team-vision vision-classifier-model-x-default-x-team-vision 1/1 Running 0 14s 10.1.171.145 brev-8dq0cch1j
Three pods, three different vcluster-team-* namespaces, all running on the same physical node brev-8dq0cch1j.
Each team sees only their own pod from inside their vCluster:
vcluster connect team-nlp -- kubectl get pods -o wide
vcluster connect team-vision -- kubectl get pods -o wide
vcluster connect team-recommender -- kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nlp-sentiment-model 1/1 Running 0 3m11s 10.1.171.180 brev-8dq0cch1j <none> <none>
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
vision-classifier-model 1/1 Running 0 3m59s 10.1.171.145 brev-8dq0cch1j <none> <none>
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
recommender-model 1/1 Running 0 2m45s 10.1.171.187 brev-8dq0cch1j <none> <none>
Finally, the queue confirms the allocation. You can check all three at the same time:
kubectl describe queue team-nlp | grep -A4 Status
kubectl describe queue team-vision | grep -A4 Status
kubectl describe queue team-recommender | grep -A4 Status
You will see the same result for all:
Status:
Allocated:
nvidia.com/gpu: 330m
Requested:
nvidia.com/gpu: 330m
Note that KAI Scheduler handles scheduling—which pods land on which GPU and in what proportion. It does not enforce GPU memory isolation at the hardware level when GPU sharing is used. Applications need to respect their memory amount (for example, setting –gpu-memory-utilization in vLLM).
Under the hood, the GPU time-slices between CUDA contexts from each pod at kernel boundaries. For hard memory isolation on supported hardware, NVIDIA Multi-Instance GPU (MIG) provides hardware-level partitioning, which can be scheduled by KAI Scheduler as well.
Get started running isolated tenant Kubernetes clusters
KAI Scheduler decides fairly who gets GPU slices and schedules them as a group using GPU sharing and DRA Driver support, hierarchical queues with guaranteed quotas plus over-quota capabilities, gang scheduling, and topology awareness. Once AI workload scheduling is solved, teams want their own clusters. vCluster gives each team its own isolated Kubernetes control plane without the cost of separate infrastructure.
Together, KAI Scheduler and vCluster provide a dedicated cluster experience for three teams with zero waste on a single GPU. The answer isn’t always more GPUs, but better utilization of infrastructure.
Ready to get started? Check out KAI Scheduler, vCluster, and NVIDIA GPU Operator on GitHub.
Learn more about KAI Scheduler and vCluster integration at KubeCon 2026 North America, November 9-12.
