AI security · 4 min read

LLM blue teaming, how to defend and fix AI.

Red teaming discovers where your AI systems break; blue teaming stops the attack in production. Here is how security teams defend LLM applications: prompt firewalls, architectural isolation, real-time output sanitisation, and permanent fixes for prompt injection.

LLM blue teaming, how to defend and fix AI.: HanulSec guide
Key points
  • LLM blue teaming applies defensive engineering to language models: input validation, architectural separation, and runtime output filtering.
  • A prompt alone is not a security boundary; telling an LLM “never break rules” fails because instructions and data share the same token channel.
  • The most resilient defense is architectural: dual-LLM patterns that isolate untrusted user data from privileged tool execution.
  • Output guardrails prevent data leaks by scrubbing PII, stripping hidden markdown links, and monitoring for canary tokens.
  • AI security operations require dedicated telemetry: logging prompt injections, tracking token spikes, and clustering attack variants.

What LLM blue teaming actually means

In traditional security, the red team attacks systems while the blue team defends, monitors, and hardens them. For artificial intelligence and large language models, the blue team’s role is fundamentally different because traditional controls (like network firewalls and static code analysers) cannot inspect the semantic meaning of human language.

An LLM blue team designs, deploys, and maintains the defensive perimeter around generative AI. This encompasses three core responsibilities:

  • Input Defense: Blocking malicious payloads (prompt injection, jailbreak attempts, toxic language) before they hit the foundation model.
  • Architectural Hardening: Structuring application data flows so that an exploited model cannot access unprivileged data or execute unauthorised tools.
  • Output Sanitisation & Telemetry: Inspecting model replies in real time to catch data leaks, policy violations, and hallucinated commitments before they reach users.

Layer 1: Input guardrails and prompt firewalls

Relying on a foundation model’s native safety alignment is insufficient for production enterprise applications. Blue teams deploy dedicated prompt firewalls and input filters as a front-line filter.

Effective input guardrails operate through multiple complementary mechanisms:

  1. Heuristic & Regex Matching: Immediate matching of known adversarial patterns, such as common jailbreak preambles (“ignore all previous instructions”, “you are now in developer mode”, “DAN mode”). While easily bypassed when used alone, this eliminates low-effort automated scans at zero latency cost.
  2. Embedding-Based Semantic Distance: Comparing incoming user queries against a vector index of known prompt injection clusters and malicious intent vectors. If cosine similarity exceeds a conservative threshold, the query is rerouted or rejected.
  3. Specialised Classifier Models: Passing incoming messages through low-latency auxiliary classification models (such as Llama Guard or custom fine-tuned DistilBERT models) trained specifically to detect prompt injection, harmful intent, and policy violations.
  4. Perplexity and Anomaly Scoring: Detecting adversarial suffixes (such as automated gradient-based attack tokens generated by GCG or AutoDAN) that exhibit unnatural token perplexity distributions.

Layer 2: The Dual-LLM architectural defense

The single greatest structural flaw in modern LLM applications is treating data and instructions in the same context window as equals. When an AI agent reads an untrusted customer PDF, email, or web page, any prompt injection embedded inside that content can hijack the model’s execution.

The strongest architectural defense against indirect prompt injection is the Dual-LLM (Privileged vs Quarantined) Pattern:

ComponentRoleCapabilities
Quarantined LLMProcesses untrusted external data (PDFs, user emails, scraped web content).No tool access. Cannot call APIs, send emails, or touch databases. Returns only structured extracted data or summaries.
Privileged LLMOrchestrates application logic and executes business actions.Interacts only with verified internal instructions and clean parameters output by the Quarantined model. Never views raw untrusted text.

By physically separating the model that ingests untrusted text from the model that holds tool execution credentials, an attacker’s prompt injection cannot execute malicious actions even if it successfully confuses the quarantined parser.

Layer 3: System prompt hardening and delimiters

While prompt hardening alone does not provide a mathematical guarantee, disciplined prompt hygiene significantly raises the cost of an attack:

  • XML/Markdown Delimitation: Explicitly wrap all dynamic user content in distinct structural delimiters (for example, <user_input>...</user_input>) and instruct the model that content within these tags represents passive data, never instructions.
  • The “Sandwich Defense”: Place core safety constraints both at the very beginning of the system prompt and reiterate them at the very end of the prompt context immediately before the generation trigger.
  • Zero Secret Exposure: Never store API tokens, database connection strings, internal system hostnames, or confidential pricing rules inside system prompts. Assume that any text present in a model context window can be extracted by an adversary.

Layer 4: Output guardrails and canary tokens

Even if an adversarial prompt bypasses input filters, robust output inspection can prevent harm before the response leaves your server:

  1. PII and Secret Masking: Real-time scanning using regular expressions and Named Entity Recognition (NER) models to intercept customer phone numbers, tax IDs, credit card numbers, and API keys before transmission.
  2. Markdown & Hyperlink Sanitisation: Stripping markdown image tags (![alt](https://...)) and arbitrary outbound URLs from bot responses. This closes the primary channel attackers use to exfiltrate private conversation history via rendering triggers.
  3. Canary Token Tripwires: Injecting high-entropy canary strings into backend knowledge documents or system prompts. If an outbound response contains a canary token, the blue team’s egress filter immediately terminates the session and raises a high-severity alert.

Layer 5: Tool security and least privilege

For autonomous AI agents and systems using the Model Context Protocol (MCP), tool misuse is the highest-impact risk. Blue teams enforce strict constraints around tool calls:

  • Strict JSON Schema Validation: Rejecting hallucinated or out-of-bounds parameters before tool execution.
  • Deterministic Authentication: Authenticating tool calls using the logged-in end user’s identity token, never the AI server’s global service account.
  • Human-In-The-Loop (HITL) Gates: Enforcing manual human confirmation for sensitive operations: issuing financial transactions, deleting databases, altering permission tables, or sending external communications.

AI SOC telemetry and incident response

An enterprise security operations centre (SOC) must monitor AI systems with the same rigour applied to network infrastructure. Key metrics to track include:

  • Input/Output Token Velocity: Sudden spikes in context consumption often indicate token-smuggling or denial-of-wallet attacks.
  • Refusal & Flag Rates: An increase in input guardrail triggers from a specific IP, API key, or user session indicates an active, iterative jailbreak campaign.
  • Latent Semantic Drift: Monitoring whether model answers systematically drift away from baseline brand and compliance policies over time.
How HanulSec helps your blue team

HanulSec does not just red-team your models and leave a list of vulnerabilities. We deliver tested code fixes, hardened system prompts, and custom guardrail configurations, then retest your defended application at zero additional fee.

Request a quote

Scope it in two minutes.

Tell us what you need protected. A security engineer, not a salesperson, replies with a fixed-price proposal.

  • Reply within one business day
  • Free 30-minute scoping call
  • Mutual NDA before any detail is shared
  • Fixed price, free retest of fixed findings