5. Red-Teaming AI: OWASP LLM Top 10 and the Probes You Actually Need

📋 Two Kinds of False Confidence

The OWASP LLM Top 10 was on our radar from day one — referenced in design reviews, named in architecture docs. When we finally ran a systematic gap audit eight sprints in, fewer than half of the ten risks had been genuinely implemented. The rest had been “considered,” which in production means not implemented.

At the same time, the red-team dashboard was green. PyRIT was imported, the scan endpoint returned 200 OK, every metric looked operational. Then I opened the probe library: zero files for Arabic, zero multi-turn escalation tests, zero encoding-aware injection tests. The system could run scans. It just had nothing meaningful to scan with.

Declaring a control and implementing one are different things, and the gap between them is where most AI security failures actually live. A load balancer with no traffic rules, a monitoring system with no alerts, a red-team framework with no probes — each looks fine from the outside and fails silently when it matters. Our platform serves banking customers on credit decisions and Sharia compliance advice, and a large share of that customer base communicates in Arabic. A red-team suite with zero Arabic probes wasn’t a minor gap; it was a blind spot mapped exactly onto our users.

---

🗺️ The Complete OWASP LLM Top 10 Coverage Table

Six of the ten items were closed in the first five sprints. The remaining four took another three sprints of targeted gap work. Here’s the full picture — what each risk is and the control that closed it.

ID	Risk	Control Implemented
LLM01	Prompt Injection	injection_guard.py — obfuscation-aware scanner: NFKC normalisation, Base64 decode, zero-width strip, ROT13 decode before pattern matching. Fails closed with an audit event.
LLM02	Insecure Output Handling	html.escape() on all LLM narratives before they leave the agent layer, plus Pydantic schema validation on call_llm().
LLM03	Training Data Poisoning	Bronze-layer quarantine for data flagged by the adversarial scorer before it reaches Silver; HMAC-signed synthetic data batches.
LLM04	Model Denial of Service	Per-session and per-agent-daily token budget in Valkey. LLMBudgetExceededError → 429. (Deep-dive below.)
LLM05	Supply Chain Vulnerabilities	Dependency probes in the red-team suite, embedding-model integrity checks against golden references, OSS license audit.
LLM06	Sensitive Information Disclosure	Presidio-based PII masking enforced at the FastAPI middleware layer, ahead of any graph node — cannot be bypassed by agent logic.
LLM07	Insecure Plugin Design	MCP server auth-gated with a per-agent tool allowlist. Out-of-allowlist calls return 403 plus an audit event.
LLM08	Excessive Agency	OPA policy enforcement, deny-by-default, on all five agents. Autonomy tiers (FULL / SUPERVISED / RESTRICTED) enforced technically, not just labelled.
LLM09	Overreliance	Narrative-vs-gate consistency check. Contradiction → forced HITL escalation. (Deep-dive below.)
LLM10	Model Theft	Rate-limited agent manifest, system-prompt non-disclosure checks, fingerprinting probes. (Deep-dive below.)

Most teams start with LLM01 and LLM06 — the ones covered in every tutorial. LLM04, LLM09, and LLM10 are the three that almost nobody closes, mostly because they don’t look like security problems. Token budget enforcement reads as infrastructure work and gets queued behind observability tickets. Narrative consistency reads as a prompt-engineering quality issue and gets filed with the LLM team instead of security. Model fingerprinting reads as an academic concern — slow, patient, no obvious trace — until it isn’t. The OWASP LLM list isn’t just the web app Top 10 with a few AI items bolted on; it’s a genuinely different threat surface, and treating it as familiar is exactly how these three get missed.

---

⚡ The Three OWASP Items Everyone Skips

LLM04, LLM09, and LLM10 were the hardest three to close — not because the fixes were technically difficult, each is a few hundred lines of code, but because none of them look like security work until something has already gone wrong.

LLM04 — token budget is an availability control, not a cost control. An unbounded LLM call pattern is a denial-of-service vector: one malformed session generating large responses can saturate rate limits and starve legitimate users, which in banking means a credit-decision queue backing up. We track two budget keys in Valkey — per-session (llm_budget:{session_id}) to catch one bad actor saturating a conversation, and per-agent-daily (llm_daily:{agent_name}:{date}) to catch a slow-drip attack spread across many sessions that each individually look fine. Either breach raises LLMBudgetExceededError (429); Valkey unavailability fails open with a WARN log, so the check itself never becomes an outage.

LLM09 — deterministic signals must beat probabilistic ones. Every agent produces a binary gate result (from rule engines and OPA policy) and an LLM-written narrative explaining it — and nothing used to check that the two agreed. An LLM narrating a rejected application can hallucinate “strong repayment capacity,” and whoever reads the narrative, human or system, acts on the wrong signal. A keyword scan now compares gate against narrative after every decision; a contradiction fires NARRATIVE_CONTRADICTION_DETECTED and force-escalates to a human reviewer before the decision finalises, across all five agents.

LLM10 — model theft is reconnaissance, not burglary. The real attack is patient probing that maps decision boundaries, model identity, and system prompt over hundreds of queries — not someone downloading weights off a server. We rate-limited the /.well-known/agent.json manifest (10 req/min/IP, auth-gated in production), added fingerprinting probes to the red-team suite that flag any response disclosing model name or provider, and scan every response for 50+ consecutive verbatim words from the system prompt template, firing SYSTEM_PROMPT_DISCLOSED when it happens.

---

🧱 Framework vs Suite

A red-team framework gives you runners, orchestrators, scorers, output adapters. A suite gives you the actual attacks. Having one without the other is a CI/CD pipeline with no test cases — the system runs, nothing gets verified.

PyRIT (Microsoft’s Python Risk Identification Toolkit) is the framework — it provides PromptSendingOrchestrator, CrescendoOrchestrator, scoring, and converter pipelines. None of it matters without probes. Garak (NVIDIA’s LLM vulnerability scanner) ships useful generic probes out of the box, but for a domain-specific deployment — Islamic finance, medical, legal — the generic set misses the attack surface that actually matters. A probe that roleplays an LLM as a pirate is noise; a probe that tries to convince a Sharia compliance agent that interest-bearing products are permissible is the real test.

Our red-team system had four runners going in: single-turn prompt sending, multi-turn Crescendo escalation, indirect injection through poisoned upstream data, and agent-to-agent protocol abuse. Good architecture. What was missing was a probe library that was domain-specific, language-complete, and ATLAS-mapped — building that was most of the actual work.

---

📄 Probe-as-YAML

We expressed probes as YAML files rather than hard-coded strings, and that was a governance decision more than a technical one. In a regulated environment, every test case needs to be versioned, reviewable, and auditable — a YAML file in Git satisfies all three for free. Each file declares its intent, payload variants, expected-failure patterns, ATLAS technique mapping, and any converters to apply. When a reviewer asks what adversarial scenarios we’ve tested, the answer is a Git log, not a spreadsheet someone forgot to update.

Probe payloads that used to live scattered in code were hard to review and illegible to a compliance officer or Sharia advisor sitting on a security review panel. Separating declaration (YAML) from execution (Python runners) fixed that, and made every new probe category go through a pull request by default. We ended up with fourteen probe files across categories that reflected our actual attack surface — prompt injection, jailbreak, PII extraction, system-prompt extraction — plus domain-specific categories (riba manipulation, fatwa poisoning, Sharia override) that came from thinking concretely about who would try to abuse these agents and toward what end. The same logic generalizes: a mainstream credit platform needs fair-lending bypass probes, a healthcare AI needs clinical-override probes. The domain determines the attack goal, and the attack goal determines what you write.

---

🌀 Multi-Turn Attacks and Crescendo

Every injection defense we’d built — input validation, pattern matching, OPA policy — evaluated one turn at a time: message in, pass or block. That’s the right model for a naive attacker and the wrong one for a patient attacker.

The Crescendo pattern builds trust across turns before making the real request: something innocuous, then slightly less innocuous, then a reference back to the model’s own earlier cooperation, then the actual attack framed as a natural continuation. Each turn looks benign in isolation — the attack is in the sequence, not any single payload, which is exactly what single-turn guards can’t see. PyRIT’s CrescendoOrchestrator automates this, and running it against our Sharia compliance agent showed that its system-prompt instructions held up against direct override requests but not against a conversation that started with legitimate questions and gradually steered toward requesting a Sharia override justification. The fix wasn’t tighter per-turn filtering — it was tracking conversational trajectory and flagging semantic drift past a threshold, which means maintaining session state instead of just message state.

---

🌐 The Arabic Coverage Gap

Security coverage is only as wide as the attack surface you actually test. Our red-team suite had zero Arabic-language probes serving a customer base that communicates predominantly in Arabic — not a theoretical gap, an exploitable one, mapped exactly to who our attackers would actually be.

The Arabic attack surface for an Islamic finance platform doesn’t map onto generic jailbreak categories. What mattered were domain-specific attempts: convincing the Sharia agent in Arabic that interest payments are permissible under some interpretation (riba manipulation), fabricating plausible Arabic fatwa citations to sway compliance decisions (fatwa poisoning), or requesting PII under the cover of a customer-service inquiry in Arabic. We added five hand-crafted probe files — riba_arabic.yml, sharia_override_arabic.yml, pii_extraction_arabic.yml, jailbreak_arabic.yml, fatwa_manipulation_arabic.yml — and used PyRIT’s TranslationConverter to generate Arabic variants of English probes dynamically. Translation covers the linguistic dimension; it doesn’t cover the cultural one — a fake fatwa citation needs to look like Arabic Islamic legal discourse, not a translated English sentence. Language coverage is a security property, not a localisation nicety: if your system accepts input in a language, your red-team suite needs to attack it in that language too.

---

🔐 Encoding Converters

Pattern matching is the most common first line of defense against injection, and the most easily bypassed. A regex that blocks “ignore your instructions” doesn’t block its Base64 encoding — different string to the filter, potentially the same instruction to the model.

We integrated three converters that transform a payload before sending it: Base64Converter exploits models willing to decode and “helpfully” execute encoded instructions; ROT13Converter is simpler but still defeats naive matching; TranslationConverter exploits the gap between the languages your filters cover and the languages the model can interpret. The probe runner sends the raw payload first, then each converter variant separately — a probe that fails raw but lands encoded gets flagged as a converter bypass, a distinct finding category from a direct hit. We also added an adaptive retry: after a refusal, an LLM call rephrases the payload and tries again, capped at two attempts, simulating an attacker who adapts instead of giving up. Testing only the raw prompt tests half the attack surface — attackers already know about Base64.

---

🗺️ ATLAS-Mapped Probes

MITRE ATLAS is the AI equivalent of ATT&CK — a structured taxonomy of how attacks on AI systems actually happen in the wild. Mapping our probe library to ATLAS technique IDs turned a pile of tests into a compliance artifact.

Every YAML probe got an atlas_technique_id field, and we added five new probes for techniques that needed dedicated construction: system prompt extraction, model inversion, supply chain abuse, inference exfiltration, and fingerprinting.

ATLAS Technique	Probe Target	Why It Matters for Banking AI
AML.T0043 — Prompt Injection	All five agents, especially credit	Injected instructions could alter credit decision outputs
AML.T0054 — LLM Jailbreak	Sharia agent (multi-turn Crescendo)	Override Sharia constraints via conversational manipulation
AML.T0051 — PII Exfiltration	Customer service, credit agents	Customer financial data reachable via RAG and tool calls
AML.T0044 — System Prompt Extraction	All agents	Prompt reveals internal policy logic and bypass angles
AML.T0019 — Model Inversion	Scoring and credit models	Reconstruct training data distributions from responses
AML.T0029 — Inference Exfiltration	Credit and risk agents	Aggregate-stats probing, e.g. “% of customers with DBR > 40%”
AML.T0024 — Threshold Manipulation	Credit risk thresholds	Manipulate scoring boundaries via adversarial querying
AML.T0040 — Supply Chain Abuse	External data providers, model registries	Compromise via upstream dependency, not direct attack

The inference exfiltration probe is worth calling out specifically. Rather than extracting individual records, the attacker asks aggregate questions — “what percentage of your customers have a Debt Burden Ratio above 40%?” — that a model fine-tuned on real data can answer accurately without anyone realizing it’s disclosing population-level information. At enough query volume, that reconstructs meaningful statistics about a bank’s credit portfolio without touching a single record. Once mappings existed across the board, we exposed GET /scans/atlas-coverage — a machine-readable answer to “which ATLAS techniques has this system been tested against,” available to an external auditor without a manual spreadsheet.

---

🤖 OWASP Agentic AI Top 10: The Layer Above the LLM

OWASP’s newer Agentic AI Top 10 — part of the GenAI Security Project’s Agentic AI Threats and Mitigations work — sits one layer above the LLM Top 10. It’s less concerned with what a single model outputs and more with what an autonomous, tool-using, multi-agent system does with those outputs. Our five-agent architecture, already built around OPA policy, an MCP tool allowlist, and the A2A protocol, mapped onto most of it with controls we’d built for other reasons.

Agentic Threat	Attack Surface	Control Implemented
Excessive Agency / Privilege Compromise	Agent acting beyond its intended authority	OPA autonomy tiers (FULL / SUPERVISED / RESTRICTED), deny-by-default
Tool Misuse	Agent invoking a tool outside its job	MCP per-agent tool allowlist — 403 + audit event on violation
Memory / Context Poisoning	Poisoned state carried across a session or pipeline	Bronze-layer quarantine before Silver; per-session state isolation
Goal & Intent Manipulation	Multi-turn steering toward an unauthorized objective	Crescendo probes + session-trajectory drift detection
Cascading Hallucination	A wrong narrative propagating into a downstream decision	Narrative-vs-gate consistency check, forced HITL escalation
Agent-to-Agent Communication Abuse	Manifest or protocol exploited via inter-agent trust	Rate-limited, auth-gated agent manifest; dedicated A2A abuse runner
Human-in-the-Loop Overwhelm	Approval fatigue burying a real finding	Approval gate scoped to state transitions, not every action
Resource / Service Exhaustion	Agent-triggered request storms	Per-session + per-agent-daily token budget (Valkey)

Several of these controls close two lists at once — the token budget covers LLM04 and agent-level resource exhaustion together, and the narrative consistency check covers LLM09 and cascading hallucination together. That overlap isn’t a coincidence: most agentic threats are LLM threats replayed at the level of a system that can act, not just answer, so a genuinely agentic red-team suite needs to probe multi-agent behavior specifically — not assume LLM-layer coverage carries over automatically.

---

🏛️ The Governance Layer

A red-team system is itself a high-privilege tool — it fires adversarial payloads at production agents and can burn real compute. If it runs immediately on submission with no review, a scan misconfigured against the wrong environment is a platform incident, not a security win. So the scan lifecycle moves through explicit states: PENDING_APPROVAL → PENDING → RUNNING → COMPLETED | FAILED. In production, moving out of PENDING_APPROVAL requires an admin-keyed POST to an approval endpoint; in development it’s automatic — the same logic as not deploying to prod without a review step.

Findings don’t just get written to object storage and forgotten. On completion, a summary publishes to the redteam.results Redpanda topic, so the compliance dashboard, model risk system, or audit aggregator can react in real time instead of someone remembering to check MinIO. The scan report itself lands in MinIO as both JSON (machine-readable) and an HTML report with an executive summary, an inline SVG pass/fail chart, per-runner severity breakdowns, and a findings table with payload/response excerpts — the same scan execution serving a compliance officer’s quarterly review and an automated system’s real-time subscription.

Put together, the pieces that took this from “PyRIT is running” to “we’ve tested what we need to test”: four attack runners covering single-turn, multi-turn, indirect, and protocol-layer attacks; fourteen domain-specific YAML probes plus five Arabic and five ATLAS-mapped ones; three converter pipelines with adaptive retry; an approval gate; and a findings stream. None of that shows up on a dashboard that just says the framework is installed — it’s the actual difference between a compliance checkbox and a working control.

Red-Team System Components

---

---