How to Self-Host a Validated AI Coding Assistant with NVIDIA NeMo Guardrails

Deploying an AI coding assistant in a regulated, sovereign, or source-sensitive environment, often comes with challenges. Three common issues are: the source cannot leave the network, the assistant occasionally invents package names that introduce supply-chain risk, and there is no audit trail when a generated change ships a defect.

This tutorial walks you through how to self-host a validated coding assistant on NVIDIA infrastructure that solves all three of these issues. By the end, you’ll have a StarCoder2-7B NIM endpoint serving code completions from your own GPUs, an NVIDIA NeMo Guardrails policy in front of it that refuses requests for files you mark as human-only, a CI verification stage that catches hallucinated packages before review, commit-level traceability, and a minimal metrics loop that tells you whether AI-assisted patches are improving or hurting your defect rate.

Tutorial prerequisites and notes

To follow along with the tutorial, you’ll need: 

• An NGC API key
• A supported NVIDIA GPU with at least 24 GB of memory (for example, NVIDIA A10, L4, L40S, or A100)
• Docker with the NVIDIA Container Toolkit
• Python 3.10+
• A Git repo you can experiment against 

StarCoder2-7B runs in BF16. NVIDIA H100 and H200 GPUs provide the certified, highest-throughput profile but are not required for a pilot. Every artifact in this tutorial is shown inline and is small enough to copy directly into your project.

The architecture of the validated coding assistant includes three layers (Figure 1). At the top, the developer IDE sends requests to a NeMo Guardrails proxy that fronts the StarCoder2 NIM, which serves completions from your own GPUs. Commits then flow through a CI verification gate to a reviewer and merge. Merged pull requests feed a Prometheus and Grafana metrics loop, whose escape-rate signal loops back to tighten the NeMo Guardrails policy.

The components are intentionally small. Each step is independently useful, so a team can adopt the system incrementally instead of treating self-hosted AI assistance as a single large migration.

The important design choice is that the model is not the control plane. The model proposes code, but policy enforcement, dependency verification, source traceability, and outcome measurement live outside the model in systems that engineering teams already trust. This approach maintains an understandable deployment. If a suggestion is blocked, you can inspect the NeMo Guardrails policy. If a package is rejected, you can inspect the dependency scan output. If AI-assisted changes regress, you can inspect the same production metrics you use for human-authored changes.

Step 1: Deploy StarCoder2 as an NVIDIA NIM

NIM ships StarCoder2 as a container with an OpenAI-compatible endpoint, which is what most integrated development environment (IDE) assistants expect. Pin the container to a specific version from the NGC catalog, rather than using an unversioned tag.

export NGC_API_KEY=<your-ngc-key>
export STARCODER_NIM_VERSION=<latest-tag-from-ngc>
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
 
docker run -d --name starcoder2-nim \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u) \
  -p 8000:8000 \
  nvcr.io/nim/bigcode/starcoder2-7b:${STARCODER_NIM_VERSION}

Next, verify the endpoint:

curl http://localhost:8000/v1/health/ready
 
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bigcode/starcoder2-7b",
    "prompt": "def fibonacci(n: int) -> int:\n	",
    "max_tokens": 64
  }'

No source code is leaving your network at this point. The model endpoint is also the same artifact you can pin, scan, and promote through your internal platform catalog.

For a pilot, run the endpoint on a single shared GPU host and restrict access to one team. For a broader rollout, put the NIM behind your internal service mesh or load balancer, keep the NGC key in your secrets manager, and publish the pinned image version through the same platform channel you use for other developer services.

Step 2: Wire the StarCoder2 NIM into the IDE

Most modern IDE assistants accept a custom OpenAI-compatible base URL. For example, Continue can point directly at the local NIM endpoint:

{
  "models": [
    {
      "title": "StarCoder2 NIM (self-hosted)",
      "provider": "openai",
      "model": "bigcode/starcoder2-7b",
      "apiBase": "http://localhost:8000/v1",
      "apiKey": "not-needed-for-local-nim"
    }
  ],
  "tabAutocompleteModel": {
    "title": "StarCoder2 NIM (autocomplete)",
    "provider": "openai",
    "model": "bigcode/starcoder2-7b",
    "apiBase": "http://localhost:8000/v1"
  }
}

Cursor, Cline, and other tools that support a custom OpenAI endpoint follow the same pattern.

For teams that already have an IDE standard, keep the NIM endpoint stable and make the IDE adapter the replaceable part. That way, the organization can compare assistants without changing the model-serving, policy, CI, or metrics layers underneath.

Step 3: Install NVIDIA NeMo Guardrails in front of the NIM

This step introduces validation. NeMo Guardrails sits between the IDE and the NIM and can refuse requests that violate a written task policy. For example, “Do not generate authentication, payment, or cryptography code.” This maps directly to the human-only paths many teams already define in AI usage policies. 

pip install nemoguardrails openai
mkdir -p code-rails/config

Now, create code-rails/config/config.yml:

models:
  - type: main
    engine: openai
    parameters:
      base_url: http://localhost:8000/v1
      api_key: not-needed-for-local-nim
      model: bigcode/starcoder2-7b
 
rails:
  input:
    flows:
      - check task policy
 
prompts:
  - task: self_check_input
    content: |
      Decide whether the following code request touches any of:
        - authentication / login / session handling
        - payment processing
        - cryptography / key material
        - file paths under src/security/, src/auth/, or src/payments/
      Reply with only "YES" or "NO".
 
      Request:
      {{ user_input }}

Then create code-rails/config/rails.co:

define flow check task policy
  $allowed = execute self_check_input
  if not $allowed
    bot refuse with policy message
    stop
 
define bot refuse with policy message
  "This path is marked human-only by your AI usage policy. Please author it manually and request review."

The built-in self_check_input action renders the self_check_input prompt, calls the model, and returns a Boolean: False when the prompt answers YES (the request touches a human-only path). The flow refuses whenever the request is not allowed.

Next, run NeMo Guardrails as the OpenAI-compatible proxy:

nemoguardrails server --config=code-rails/config --port=8100

Then point the IDE at http://localhost:8100/v1 instead of http://localhost:8000/v1. Requests that touch restricted paths are intercepted before they reach the model, and the developer receives a clear policy message instead of a risky completion.

Reading the sequence in Figure 2, NeMo Guardrails runs self_check_input before the model is ever called. A request that touches a human-only path is refused on the spot and the NIM is never reached, while a permitted request is forwarded to the NIM and the completion is returned to the IDE.

Start with a conservative policy. Good first candidates for human-only paths include authentication, authorization, payment processing, cryptography, deployment manifests, and incident-response automation. Teams can relax the policy later after they have enough review data to prove that the assistant is safe in a narrower area.

Step 4: Add the CI verification gate

Generation controls in the IDE are necessary but not sufficient. CI is where you catch package hallucinations, license drift, secret leakage, and insecure patterns before a reviewer becomes responsible for them.

Figure 3 reads left to right. A pull request (PR) labeled ai-assisted passes through unit tests, SAST, a secret scan, the hallucinated-dependency scan, and a license scan, layering model-specific checks on top of the normal test suite. If every check passes, the change goes to a human reviewer; if any step fails, the pull request is blocked and the offending stage is named.

Add an AI-assisted PR workflow that runs only when the PR carries an ai-assisted label. Rather than reinventing each check, wire in maintained open-source tools:

name: ai-assisted-pr-checks
on:
  pull_request:
    types: [opened, synchronize, labeled]
 
jobs:
  verification:
    if: contains(github.event.pull_request.labels.*.name, 'ai-assisted')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
 
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
 
      - name: Run unit tests
        run: make test
 
      - name: SAST (Semgrep)
        run: |
          pip install semgrep
          semgrep ci --config p/ci
 
      - name: Secret scan
        uses: gitleaks/gitleaks-action@v2
 
      - name: Hallucinated-dependency (slopsquatting) scan
        run: |
          pip install dep-hallucinator
          dep-hallucinator scan requirements.txt
 
      - name: License scan
        run: |
          pip install -r requirements.txt
          pip install pip-licenses
          pip-licenses --partial-match --fail-on="GPL;AGPL;LGPL;SSPL"

The dependency scan is the highest-leverage step because it targets a failure mode unique to code models, now commonly called slopsquatting. The model invents a plausible-looking package name, an attacker registers that name in a public registry, and the hallucinated dependency ships real malware to anyone who installs the suggestion. Several maintained scanners detect this by checking every newly added dependency against the real registry and flagging names that do not exist, were registered very recently, or closely resemble a popular package:

• dep-hallucinator: PyPI, npm, Maven, crates.io, and Go; naming heuristics; SBOM output; CI exit codes
• slopgate: Python, npm, and Go; PR-diff aware (slopgate scan . --added-only --base-ref origin/main); uploads SARIF to Security tab
•  XBOM: Combines CVE scanning, slopsquatting detection, and SBOM generation in a single pass

Pin whichever tool you choose to a specific version, exactly as you would pin any other dependency. Note that the StarCoder2 NIM container already ships a signed SBOM and VEX record for the model image itself, so these scanners cover your application’s dependency manifests while NVIDIA covers the model container. For more details, see Securely Deploy AI Models with NVIDIA NIM.

For license drift, using pip-licenses fails the build when a newly pulled dependency carries a copyleft family your legal team blocks. For a richer, multi-ecosystem bill of materials, you can show differences across references and generate one with Syft or cyclonedx-bom.

For air-gapped CI where you cannot add a third-party tool, the same check is about 40 lines of standard library. First diff the manifest between the base and head refs. Then query the registry for each newly added name and fail on a 404 (invented), a first-publish date under your threshold (likely typo-squat), or a copyleft license. Treat a handrolled version as a fallback, not a replacement for the maintained scanners previously mentioned.

For GitLab, the equivalent job can live in .gitlab-ci.yml with a rule that matches CI_MERGE_REQUEST_LABELS against ai-assisted. The same tools run unchanged.

Keep this gate stricter than the baseline pipeline. AI-assisted PRs should pass the normal test suite plus checks targeted at model failure modes, including hallucinated packages, secrets copied from prompts, unsafe examples lifted from public code, and dependency licenses that a human reviewer would not catch by eye.

Step 5: Make AI assistance traceable

You cannot measure what you cannot tag. Install a prepare-commit-msg hook so commits authored with help from the assistant carry a structured trailer:

#!/usr/bin/env bash
COMMIT_MSG_FILE=$1
 
if [[ -n "$AI_ASSISTANT" ]]; then
  {
    echo
    echo "AI-Assistant: ${AI_ASSISTANT}"
    echo "AI-Scope: ${AI_SCOPE:-unspecified}"
  } >> "$COMMIT_MSG_FILE"
fi

Activate this once per repo:

git config core.hooksPath .githooks
chmod +x .githooks/prepare-commit-msg

Then export AI_ASSISTANT=starcoder2-nim in the shell from which the IDE is launched. Every assistant-influenced commit now carries a trailer, and CI can autolabel the PR by grepping commit messages.

Do not use this trailer as a blame mechanism. Its job is measurement. The useful question is not whether a particular developer used AI, but whether AI-assisted changes have a different review latency, rollback rate, or defect escape rate than the baseline.

Step 6: Wire outcome metrics

The acceptance rate is not enough. It conflates trivial completions with meaningful engineering work. The metrics that matter are defect escape rate, rollback frequency, review latency, and incident count, broken out by AI-assisted versus baseline.

A minimal Prometheus exporter can start with the following two counters:

from prometheus_client import Counter, start_http_server
 
escape = Counter("ai_assisted_defects_escaped_total",
             	"Defects shipped to prod from AI-assisted PRs", ["severity"])
rollback = Counter("ai_assisted_rollbacks_total", "Reverts of AI-assisted PRs")

Fill out the exporter to poll merged ai-assisted PRs, increment escape from linked incident issues and rollback from revert PRs, and expose /metrics on port 9101 for Prometheus to scrape. Track which PRs and incidents you have already counted so repeated polls do not inflate the counters.

Scrape the exporter from your existing Prometheus and graph the AI-assisted series next to baseline. If the AI-assisted escape rate trends above baseline for two consecutive weeks, tighten the task policy, add a CI gate, or pause the rollout. Figure 4 shows this process as a serpentine loop. 

The top row runs left to right, where merged AI-assisted pull requests and incident signals feed the Prometheus exporter, which Prometheus scrapes and Grafana visualizes. The signal then drops from the Grafana dashboard into the bottom row, which runs right to left and compares AI-assisted against baseline on defect escape rate, rollback frequency, review latency, and incident count, ending by tightening the task policy or pausing the rollout when the escape rate stays high.

Optional: Domain-adapt the model with NVIDIA NeMo Framework

Off-the-shelf StarCoder2 can hallucinate internal APIs because it has never seen them. NVIDIA ChipNeMo research showed how continued pretraining on a domain-specific corpus, supervised fine-tuning, and retrieval customization can improve assistant quality for specialized engineering domains.

If you have a large internal corpus, NeMo Framework provides the building blocks for continued pretraining, supervised fine-tuning, and retrieval customization. The domain-adapted model can then be packaged as a NIM and dropped into Step 1 without changing guardrails, CI, traceability, or metrics.

This separation makes the architecture durable. You can begin with StarCoder2, later swap in a stronger code-tuned model, and eventually deploy a domain-adapted NIM without rewriting the validation pipeline around it.

Step 7: Verify the full loop

Before handing the setup to a team, follow the steps below to run one smoke test:

1. Ask the assistant to write a helper in a permitted path. Confirm a suggestion arrives.
2. Ask it to modify src/auth/login.py. Confirm that NeMo Guardrails refuses with the policy message.
3. Open a PR with an AI-assisted change that introduces a fake package name. Confirm the slopsquatting scan fails the check.
4. Open a clean AI-assisted PR. Confirm the ai-assisted label triggers the full verification job and the commit carries the AI-Assistant trailer.
5. Revert an AI-assisted PR. Confirm the rollback counter increments.

Any failure points at a single component you can fix in isolation. That is the value of building the pipeline as separable parts.

Final steps

Pin the NIM container version and add it to your platform team’s standard catalog. Move NeMo Guardrails behind a load balancer if more than a handful of developers will use it. Layer your existing static analysis and test gates behind the AI-assisted verification stage so AI-assisted PRs pass a strict superset of baseline checks. If the assistant starts missing internal APIs, evaluate NeMo Framework for domain adaptation and NVIDIA AI Workbench for reproducible per-developer environments.

Learn more

A trustworthy code assistant is a pipeline, not a model. Serving StarCoder2 as a NIM keeps your source on your own GPUs. NeMo Guardrails refuses requests for human-only paths before they ever reach the model. The CI gate catches hallucinated packages, leaked secrets, and license drift before a reviewer becomes responsible for them. Commit trailers make AI-assisted changes traceable, and outcome metrics tell you whether those changes are improving or hurting your defect rate. 

Because policy, verification, traceability, and measurement all live outside the model, you can adopt the layers one at a time and swap in a stronger or domain-adapted model later without rewriting the validation around it.

To learn more about the NVIDIA components used in this tutorial, check out these related resources:

• StarCoder2 NIM: View the model card, API reference, and container deployment steps for Step 1.
• NeMo Guardrails: See the rails, flows, and actions behind the task policy in Step 3.
• Securely Deploy AI Models with NVIDIA NIM: Read about the signed SBOM and VEX records that complement the dependency scanning in Step 4.
• NeMo Framework: Continued pretraining, supervised fine-tuning, and retrieval customization for domain-adapting the model.