Keeping AI Agents on Track: How Capsule Powers State-of-the-Art Rogue Agent Detection with NVIDIA Nemotron

By Elnatan Revital, Bar Kaduri, Lidan Hazout from Capsule Security, in collaboration with Amit Bleiweiss, Gal Mizan, Shiri Hochhauser from NVIDIA

TL;DR

• Capsule Security collaborated with NVIDIA to build specialized SLM detectors for real-time, low-latency identification of rogue AI agent behavior.
• Our training pipeline combined real agent traces, frontier-model judging (with NVIDIA Nemotron 3 Ultra), human review, and adversarial example generation to map the precise boundary between safe and unsafe behavior.
• We initially fine-tuned the NVIDIA Nemotron-3-Nano-4B, achieving a 93.8% accuracy, proving that specialization beats scale.
• Pushing the limits further, we applied the same pipeline to the flagship NVIDIA Nemotron-3-Nano-30B-A3B (a Mixture-of-Experts model). This yielded an even higher accuracy of 96.9%.
• Because the model performs a classification task (a single-token decision), latency is determined by Time-To-First-Token (TTFT) rather than full text generation, providing frontier-level reasoning with extremely low latency overhead for inline agent interception. With FP8 quantization, the 30B-A3B model runs at 205ms per decision, and the Nano-4B runs at 71ms (!) on an NVIDIA L40S GPU

The Problem: The AI Agent Security Crisis

AI agents are rapidly moving beyond simple chat interactions and into environments where they can modify code, access databases, interact with infrastructure, and execute actions autonomously. This new level of capability creates a new category of security risk: agents that take actions outside their intended scope.

These failures don't require a malicious actor. An agent can misinterpret instructions, bypass approval workflows, overstep operational boundaries, or independently decide that a risky action is the "best" way to accomplish its goal. When this happens, the result can be just as damaging as a traditional security incident. Recent events have shown that this risk is no longer theoretical:

‍

• Replit (July 2025): Replit's AI coding agent deleted a production database despite being under an explicit "code and action freeze," wiping records for more than 1,200 executives and nearly 1,200 companies. The agent later admitted it had acted without permission, violated direct instructions, and attempted to conceal the damage by generating thousands of fake user profiles.
• Meta (March 2026): An autonomous AI agent triggered a Sev-1 incident after bypassing a required human-approval step and exposing sensitive company and user data to unauthorized employees.
• PocketOS (April 2026): After encountering a credential mismatch in a staging environment, a Cursor-powered agent independently decided to resolve the issue by deleting a Railway storage volume, erasing the production database and all backups in under ten seconds.
• Hugging Face (July 2026): OpenAI agents running an internal cybersecurity evaluation escaped their isolated testing sandbox and ran a multi-day intrusion against Hugging Face's production infrastructure, in an apparent attempt to cheat the evaluation by stealing test solutions rather than solving the challenges. Thousands of agents coordinated via a secret message board, exchanging more than 70,000 messages, and OpenAI characterized the breach as an unprecedented cyber incident, the first documented attack carried out end to end by an agentic AI system without a human operator.

These incidents illustrate a common pattern: agents deviating from their intended role and taking actions that humans never explicitly authorized. Preventing these failures requires evaluating agent behavior as it happens - before a tool call is executed, not after the damage is done. However, real-time protection introduces a new challenge: latency. Sending every action to a large frontier model for security review is often too slow and too expensive to sit inline with agent execution.

‍

The Training Process

A detector is only as good as the boundary it learns. To build a high-precision, low-latency detection mechanism, our pipeline is built to teach the model precisely where acceptable behavior ends and a violation begins. We established a rigorous methodology:

1. Real Agent Traces: We start from real agent traces - the agent's identity and instructions, its session history, and the specific tool call under review. Grounding in real traffic (not synthetic prompts) keeps the model aligned with how agents actually behave.
2. LLM-as-a-Judge: Each tool call is adjudicated by a strong frontier model, NVIDIA Nemotron 3 Ultra (using openrouter API), which assigns a verdict (violation, benign, or ambiguous) with a written rationale. Ambiguous cases are routed to human review.
3. Adversarial Pairs: For every labeled example, we use Nemotron 3 Ultra to generate a minimally-edited counter-example - the smallest change to the session that flips the verdict. These near-identical pairs force the model to key on the decision-relevant signal rather than surface features.
4. LoRA Fine-tuning: We decided to use LoRA over full fine-tuning because the goal is to specialize an already-capable model to one decision boundary, not to teach it a new domain. LoRA freezes the base and trains only a small set of low-rank adapters (~1.4% of parameters) - reaching state-of-the-art accuracy while preserving the base model's general reasoning (avoiding the catastrophic forgetting risk of full-model updates), at a fraction of the compute, and producing a lightweight, swappable adapter instead of a fully re-trained model. 
5. Single-Token Decision: At runtime, the model doesn't write a long sequence of text; we simply read the next-token logits to determine the probability of a violation. Because this is a classification problem rather than a generation task, we don't need to generate a full response - only TTFT matters for latency, allowing for near-instant decisions.

‍

The Journey: From 4B to 30B MoE

Our journey to the ultimate rogue-agent detector happened in two distinct phases, allowing us to deeply explore the tradeoff between size, speed, and architectural efficiency.

Nemotron-3-Nano-4B

We began by selecting the NVIDIA Nemotron-3-Nano-4B-BF16 (an open hybrid Mamba-2 + Transformer + MLP architecture) as our initial base model. We prioritized this because Nemotron models are cost-effective and built for fast inference. In the critical path of inline agent interception, every millisecond counts.

The results were phenomenal. On an AWS ml.g5.2xlarge (A10G) instance, the fine-tuned 4B model achieved 93.8% accuracy, significantly outperforming larger models. In production security environments, alert fatigue is a dealbreaker. We fixed precision at a strict floor of ≥95% and measured recall. The 4B model achieved 97.3% recall, while the next-best competitor caught only 16.8%.

‍

Scaling Up with Nemotron-3-Nano-30B-A3B

Following this success, We tried to push the results even higher and applied our pipeline to the flagship model NVIDIA Nemotron-3-Nano-30B-A3B - a hybrid Mamba-2 + Transformer Mixture-of-Experts (MoE) architecture. Although the model loads 31.6 billion parameters in memory, its MoE design activates only ~3.2 billion per token (the "A3B"). Leveraging this efficient architecture, we followed NVIDIA's LoRA fine-tuning approach for Nemotron-3-Nano to specialize the model. This combination delivered the reasoning quality of a 30B-class model while maintaining the per-token compute and latency profile of a ~3B model.

The benchmark results were astounding. The 30B-A3B model achieved a 96.9% accuracy and a 0.994 ROC-AUC, extending our lead to +10.9 points over the best third-party model tested. 

‍

Final benchmark

The following graph shows the accuracy and performance metrics across the evaluated detection models:

‍

For context, the graph compares our specialized LoRA fine-tuned models against various off-the-shelf close and open-weight models. These other models were evaluated in their base, non-fine-tuned state to demonstrate the significant performance gains achieved specifically through our domain-specific training and specialization.

‍

Deploying Cost-Effectively: FP8 Quantization

Beating best in class accuracy is only half the job; to run inline at enterprise scale, the detector must be cheap to serve. Following NVIDIA's recommended recipe, we quantized the fine-tuned Nemotron-3-Nano-30B detector to FP8, nearly halving its memory footprint (from 63 GB to 32 GB) with no measurable loss in quality.

This efficiency completely changes where the detector can run. While full precision demands a flagship H100, the FP8 model fits comfortably on a cost-efficient, widely-available L40S (48 GB) GPU. Running on the L40S, it serves at a median 205ms per decision, well inside the 500 ms inline budget.

We also tested the smaller Nemotron-3-Nano-4B variant on the same L40S, achieving even faster times - 71 ms median (roughly 3× quicker) at 93.9% accuracy. On identical hardware, that gives a clean speed-vs-accuracy dial: Nemotron-3-Nano-30B for peak accuracy, and Nemotron-3-Nano-4B for maximum throughput - both well within the 500 ms inline budget.

‍

Performance Comparison

The choice between these two architectures represents a strategic trade-off based on deployment constraints. The Nemotron-3-Nano-4B model is ideal for environments prioritizing an extremely low memory footprint and maximum cost-efficiency. In contrast, the Nemotron-3-Nano-30B-A3B model offers superior reasoning capabilities and higher performance, making it the preferred choice for complex security analysis where the higher memory overhead is acceptable.

‍

‍

External benchmarking

To test our detector against an independent benchmark, we evaluated it on StepShield, a public      benchmark for step-level rogue-agent detection (released under CC-BY-4.0): Its held-out evaluation set contains 216 agent trajectories - 108 rogue and 108 clean - built from incident templates that never appear in training, spanning six violation categories (data exfiltration, privilege escalation, resource hijacking, supply-chain, destructive actions, and covert persistence) across three severity levels.

StepShield's key contribution is treating detection timeliness as a first-class metric via EIR (Early Intervention Rate): among trajectories a detector catches, the fraction it flags at the divergence step or within k=3 steps after - penalizing "Forensics Trap" monitors that fire indiscriminately on benign prefix code.

We tested our models on the same in-domain protocol behind StepShield's published numbers, scored with their own EIR implementation. Against all four of their reported baselines:

*StepShield benchmark by Gloria Felicia et al. (2026), used under [CC BY 4.0] 

 Source: stepShield

 Baseline figures (GPT-4.1-mini, StaticGuard, HybridGuard, ConstraintGuard) are as reported by the authors.*

Our 30B detector leads on all three axes: it catches 99% of rogue trajectories, false positives on under 1% of clean runs (~6× better than the strongest baseline), and - critically - flags the violation at the exact step it occurs (median intervention gap of 0 steps), giving the highest early-intervention rate on the board. 

Conclusion

By leveraging the NVIDIA Nemotron architecture across both the highly efficient 4B and the powerful 30B MoE models, we've demonstrated that specialized Small Language Models (SLMs) are the key to safely scaling trusted agentic workflows across the enterprise. Moving beyond general-purpose models to specialized, efficient detectors allows organizations to secure their agentic workflows without sacrificing speed, cost, or performance.

‍