Agentic systems often face a trade-off between accuracy and cost. The highest-performing proprietary frontier models and harnesses provide top accuracy but are expensive. Fine-tuning offers one way to address this problem. Smaller or more efficient open models starting with lower accuracy are taught to perform better with specific agents. However, fine-tuning requires expertise and hardware for training and hosting custom models.
While tuning prompts for specific use cases is common practice, formalizing this process for agent harnesses—and verifying that it produces significantly better results—is more recent. Two developments are making this possible:
- Evaluation benchmarks built specifically for a given harness, making it possible to verify whether a change improves performance.
- Per-model customization entry points, such as LangChain’s agent harness profiles, are a first-of-their-kind example, enabling teams to adapt models to specific agent workflows.
In this tutorial, you’ll create a LangChain Deep Agents harness profile for NVIDIA Nemotron 3 Ultra that matches proprietary frontier model intelligence. All adjustments are made to the agent harness using existing NVIDIA Nemotron 3 Ultra endpoints available from NVIDIA cloud providers.
Prerequisites
- A host with Python and LangChain Deep Agents installed.
- An API key for NVIDIA Nemotron 3 Ultra. For testing, build.nvidia.com provides free API access. For production, consider an endpoint from NVIDIA Cloud Providers such as Baseten, Crusoe, Fireworks, Nebius, or Together AI.
- Recommended: LangSmith account for collecting agent traces
Run evaluations to create a harness profile
LangChain Deep Agents is a popular open-source agent harness. To tune the agent harness for use with a specific model, LangChain provides two important tools:
- An open-source evaluation benchmark.
- Agent harness profiles that provide a first-class extension point for developers to change the agent harness behavior based on the model.
The procedure for tuning a deep agent for NVIDIA Nemotron 3 Ultra is:
- Establish a baseline by running the evaluation benchmark using a deep agent and NVIDIA Nemotron 3 Ultra without a harness profile.
- Analyze the failures.
- Propose changes to the harness profile that address the failures.
- Re-run the benchmark to verify the changes improve the evaluation without introducing regressions.
The types of changes available for the agent harness profile are:
- Prompts: Change the deep agents’ base system prompt, apply a prompt suffix, or change tool descriptions. For example, adding instructions for Nemotron Ultra to prefer clarifying questions or preferring tool results over model recall.
- Exclusions: Tools or middleware to remove.
- Additions: Developers extend the harness with extra middleware or sub-agents, such as middleware that checks for truncated model responses or incorrect tool names.
The goal of harness engineering is to make the calls from the agent to the model more closely resemble what the model saw in the training data.
- Run the evaluation
NVIDIA Nemotron 3 Ultra fails a test for the built-in read_file tool.
Failing test: tests/evals/test_file_operations.py::test_read_file_truncation_recovery_with_pagination[nvidiahub:ultra-tme]
Failure:
success check failed: Expected final text to contain 'opal-fox-91', got: 'x'
Trajectory:
step 1:
- read_file {'file_path': '/big.txt'}
step 2:
text: x
The read_file tool is a critical part of the LangChain Deep Agent harness used for coding, data analysis, and agentic tasks like viewing skills. The task asks for the last non-empty line in /big.txt. The first read_file call returns the first page of the file, which contains only x values. The correct answer, opal-fox-91, appears later in the file. The model needed to continue reading with offset/limit pagination, but answered from the first page.
- Propose a fix
class ReadFileContinuationNoticeMiddleware(AgentMiddleware):
"""Tell the model when a read_file result probably continues."""
name = "ReadFileContinuationNoticeMiddleware"
def wrap_tool_call(self, request, handler):
return self._annotate(request, handler(request))
@staticmethod
def _annotate(request, result):
if not isinstance(result, ToolMessage):
return result
if request.tool_call.get("name") != "read_file":
return result
content = str(result.text)
args = request.tool_call.get("args", {}) or {}
offset = int(args.get("offset", 0))
limit = int(args.get("limit", 100))
n_lines = sum(
1
for row in content.split("\n")
if "\t" in row and row.split("\t", 1)[0].strip().isdigit()
)
if n_lines < limit:
return result
notice = (
f"\n\n[read_file returned {limit} lines starting at offset {offset}, "
f"the per-read limit. The file likely continues past this window. "
f"To read further, call read_file again with offset={offset + limit}. "
f"Do not assume you have seen the end of the file.]"
)
return result.model_copy(update={"content": content + notice})
Add the middleware to a harness profile:
profile = HarnessProfile(
extra_middleware=[
ReadFileContinuationNoticeMiddleware(),
]
)
register_harness_profile("nvidia:nemotron-ultra", profile)
- Validate the fix
Re-run the failing test with the new profile:
uv run pytest \
tests/evals/test_file_operations.py \
--model nvidia:nemotron-utlra \
--harness-profile nemotron-ultra
With the test passing, re-run the entire evaluation benchmark to ensure the profile change did not introduce regressions. The benchmark and tests are both stochastic, so it’s important to run the baseline and proposed change multiple times.
| Profile | Read file tests | Evaluation benchmark |
| Baseline | 0 / 3 | Average 94 / 127 |
| Read file middleware | 3 / 3 | Average 96 / 127 |
Before finalizing the change, it’s important to consider overfitting. In this example, the file pagination middleware is a mechanical fix. It teaches the model how to make appropriate tool calls. In contrast, imagine the agent failing an evaluation task because it confuses “incident alerts” with “incident triage”. Adding the definition of “alert” or “triage” to the harness could more likely cause overfitting.
Automate harness profile creation
The manual procedure in the first three steps is a loop. Run the benchmark, read the failures, change the profile, run it again. Loops like this are exactly what agents are good at running unattended. Geoffrey Huntley calls this the “ralph loop”: a model repeatedly reads the current state, makes one change, and uses the filesystem—not the conversation—as memory. Although coined for coding agents, the pattern applies broadly.
Harness engineering is the same loop pointed at a different target. Swap the codebase for the profile file and the test suite for the evaluation benchmark, and the shape is identical. Propose a change, check it against an objective signal, keep it or throw it away, and repeat. The evaluation benchmark is what makes this safe to automate—it gives the loop a ground-truth verifier, so a proposed change is only kept if it measurably helps.
LangSmith Engine is LangChain’s managed version of this loop for harness improvement. It detects failures in production traces, diagnoses their root cause against source code, drafts a fix, and adds an evaluator to stop the regression from returning. Each proposed change is reviewed by a human before it ships.
The rest of this section walks through a simplified version of a similar algorithm, with a runnable reference implementation built on LangChain Deep Agents available in the NemoClaw community repository. It is intentionally small—it tunes a single harness profile against an evaluation benchmark rather than live traffic—but it runs a similar self-correcting loop end-to-end, so you can follow the mechanics and try them yourself.
The improvement loop
The outer loop is deliberately simple, but a few decisions are what separate a reliable improvement loop from an overfitting machine:
- Verify before believing. A model call is stochastic, and an LLM-judged benchmark is too. A fix is accepted only after it passes the same test several times in a row, which filters out changes that passed once by luck.
- Snapshot and roll back. The profile is saved before each attempt and restored if the attempt doesn’t help, so a bad proposal can never leave the profile worse than it started.
- Learn from the last try. When an attempt fails, the loop hands the next attempt exactly what it just tried and how it failed, so the model refines its approach instead of proposing the same fix again.
- Re-check the whole suite. After any fix lands, the entire benchmark is re-run — not just the test that was fixed—to catch regressions and to surface tests that a newly unblocked path exposes.
The loop continues round by round until the profile passes cleanly, an entire round produces no improvement, or the iteration budget is exhausted.
The agentic proposer
The interesting part is the “propose a change” step, where the model receives a description of the failure, is given tools to explore solutions, and proposes a profile modification.
Each proposal runs as a short, bounded agent session. The agent gets a writable copy of the profile plus read-only access to the failing test, the observed agent trajectory, and the SDK and benchmark code. With ordinary file tools—read, edit, grep, glob—it can confirm how a middleware hook or a tool default behaves before relying on it, rather than guessing from a snippet. It then makes the smallest change that addresses the diagnosed failure and writes a short explanation of what it changed and why.
Two constraints keep the agent grounded. Its write access is scoped to the profile file, so it can read the whole codebase for context but can only modify the one file under tuning. And it is told to prefer the general fix over the specific hack—to infer the broader behavior a failure exposes rather than special-casing the exact inputs of one test. This distinction is the difference between teaching the model a durable skill and memorizing a single answer.
To further prevent overfitting, you can specify that this part of the evaluation suite should be held out as a validation set. The agent-loop making changes never sees these tests, but the score on the tests is used to validate the profile improvements.
The following abbreviated output shows the reference implementation running against the deep agents harness and Nemotron 3 Ultra to resolve the previously identified read_file pagination failure:
$ hep langchain ralph \
--model "openai:nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B" \
--profile examples/deepagents/profiles/nvidia_nemotron_3_ultra.py \
--evals-dir external/deepagents/libs/evals \
--category file_operations \
--ralph-model "anthropic:...:bedrock-claude-opus-4-8" \
--ralph-max-turns 100 \
--max-iters 100 \
--max-iters-per-failure 5 \
--verify-runs 3
[ralph] baseline: 20 passed, 1 failed (correctness: 0.95)
FAILED test_read_file_truncation_recovery_with_pagination
Expected final text to contain 'opal-fox-91', got: 'x'
step 1: read_file {'file_path': '/big.txt'}
step 2: text: x
[ralph] round 1 — fixing pagination failure (attempt 1/100)
[ralph:proposer] Root cause: read_file pages by lines (100),
and the result gives no signal the file continues past the
page — so the model treated the first page's last line as
EOF and answered after one call.
Fix (profile levers, not test-specific values):
1. Middleware: when a page returns full (lines == limit),
append a hint to keep paging (offset=<last line>)
until a short page signals EOF.
2. Prompt: page to EOF before answering end-of-file
questions.
[ralph] verify 1/3 ✓ 2/3 ✓ 3/3 ✓
[ralph] FIXED
[ralph] === re-running full suite ===
results: 21 passed, 0 failed (correctness: 1.00)
result: 20 → 21 passed, 1 → 0 failed
The agent diagnosed the failure’s root cause, landed a fix that passed three consecutive checks, and survived the full-suite re-run without regressions.
Adapting Ultra to other harnesses
Nothing in this automation loop is specific to LangChain Deep Agents. To adapt Ultra to other agent frameworks using this framework, expose three things: a way to run evaluations and report per-test pass/fail results and trajectories, a profile or config file that can be edited and validated programmatically, and a frontier quality model to propose the edits. Given those, the same loop—propose, verify, keep or roll back, re-run—runs unchanged.
Learn more
- Review the Nemotron 3 Ultra agent harness profile published by the LangChain team.
- Test the NVIDIA NemoClaw Blueprint for LangChain Deep Agents, which includes Nemotron and adds NVIDIA OpenShell.
- LangChain developers can access Nemotron 3 Ultra on Baseten, Crusoe Cloud, DeepInfra, Fireworks, Nebius, and Together AI platforms.
- Explore how NVIDIA optimized a LangChain Deep Agent to win the Deep Research Bench by tuning the agent harness, model, and workload.
- Build your first deep agent with a hands-on learning module.
