AI security · 4 min read

AI chatbot security, twenty checks before launch.

Most AI chatbot incidents come from a handful of gaps that are easy to check: what the bot can see, what it can do, and what happens to its answers. Here are twenty checks, in five groups, to run before your chatbot or AI agent talks to real customers.

AI chatbot security, twenty checks before launch.: HanulSec guide
Key points
  • Treat everything the chatbot reads, including documents and web pages, as untrusted input.
  • Give the bot the least data and the fewest tools it needs, and check the user’s permissions outside the AI, not inside the prompt.
  • Never let the AI’s answer run as code, SQL or a command without checks, and put a human in front of risky actions.
  • Log prompts, answers and tool calls so you can investigate, but mask personal data in those logs.
  • Test with real attacks before launch and after every change to the model, prompt, data or tools.

How to use this checklist

Go through it with the people who built the bot: a developer, someone who knows the data, and whoever owns the customer experience. Each check is a yes or no question. A “no” is not a disaster, but it is a gap you should close or consciously accept before launch.

The checks follow the OWASP Top 10 for LLM Applications, grouped by the five places where chatbots usually go wrong.

1. What the bot reads

  1. Is everything the bot reads treated as untrusted? That includes user messages, uploaded files, web pages, emails and support tickets. Any of them can carry hidden instructions (prompt injection).
  2. Is your system prompt free of secrets? Assume users can make the bot reveal its instructions. API keys, internal URLs, discount codes and staff names do not belong there.
  3. Are long and unusual inputs limited? Set a maximum message length and file size, and reject file types the bot does not need.
  4. Do you test in the languages your users write in? Safety rules that work in English often fail in Hindi, Arabic or mixed “Hinglish”.

2. What the bot can see

  1. Can the bot only reach data the current user is allowed to see? Check permissions in your own code before data reaches the AI. A line in the prompt saying “only show the user their own orders” is not access control.
  2. Is your document search (RAG) filtered per user? If the bot searches a knowledge base, make sure one customer’s files, or internal HR and finance documents, can never come back in another user’s answer.
  3. Is personal data kept to the minimum? Under India’s DPDP Act you need a purpose for every piece of personal data. Do not give the bot full customer records when it only needs an order status.
  4. Do you know where prompts and answers go? Check whether your AI provider stores them, for how long, in which country, and whether they are used for training.

3. What the bot can do

  1. Does the bot have only the tools it needs? Every tool, plug-in or MCP server is something an attacker can try to trigger. Remove what is not used.
  2. Are tool permissions the smallest possible? Read-only where you can. Scoped to the current user. Short-lived credentials, never an admin key.
  3. Do risky actions need a human? Refunds, payments, deleting records, sending emails outside the company and changing account details should need a person’s confirmation.
  4. Are there limits on volume and cost? Rate-limit users and cap tool calls per conversation, so one abusive user cannot run up your AI bill or flood a system.

4. What happens to its answers

  1. Is the answer treated as untrusted before it is used? Never run AI output directly as code, database queries or shell commands, and escape it before showing it in a web page, so it cannot carry a script.
  2. Can the bot only link and load from places you trust? Hidden image links and markdown links are a common way to leak data out of a chat. Block or allow-list them.
  3. Is the bot stopped from making promises? Prices, refunds, legal or medical advice and policy exceptions should come from your systems, not from the model’s imagination. In 2024 a Canadian tribunal held Air Canada to what its chatbot told a customer.
  4. Is there a clear way to reach a human? And does the bot tell users they are talking to an AI? The EU AI Act requires that from August 2026.

5. Logs, monitoring and testing

  1. Are prompts, answers and tool calls logged? You cannot investigate what you did not record. Mask personal data and keep logs for as long as your rules need (CERT-In asks for 180 days of logs in India).
  2. Do alerts fire on attack patterns? For example, repeated attempts to get the system prompt, sudden spikes in tool calls, or answers containing data that looks like card or Aadhaar numbers.
  3. Has the bot been red-teamed by people trying to break it? Automated scanners catch known attacks. People find the ones specific to your business. See what LLM red teaming is.
  4. Do you retest after every change? A new model version, prompt, data source or tool can undo earlier fixes. Retest before each release, or monthly if you ship often.
Fast start

If you only have one afternoon, do checks 5, 9, 11 and 13. Limiting what the bot can see and do, and never trusting its output, prevents the most damaging incidents even when an attacker gets the bot to misbehave.

After the checklist

A checklist tells you where the gaps are likely to be. It does not prove the bot is safe. For that you need someone to attack it the way a real attacker would, record every successful attempt and check your fixes. That is what an LLM red teaming engagement does, and its report also serves as evidence for ISO/IEC 42001, the DPDP Act and customer security reviews.

Questions

Questions buyers ask before they call us.

Is this checklist enough to make our chatbot safe?
No checklist can promise that. It closes the common gaps and shows you where to focus. Testing the bot with real attacks is what tells you whether the defences actually hold.
Does this apply if we use ChatGPT, Gemini or Claude through an API?
Yes. The AI provider secures the model. Your prompt, your data, your tools and how your app uses the answers are your responsibility, and that is where most of these checks apply.
Which checks matter most for AI agents?
The tool checks (9 to 12) and the output checks (13 and 14). Once an AI can take actions, a successful attack can move money or data, not just produce a bad answer.
How often should we go through it?
Before launch, and again whenever you change the model, the system prompt, a data source or a tool.
Request a quote

Scope it in two minutes.

Tell us what you need protected. A security engineer, not a salesperson, replies with a fixed-price proposal.

  • Reply within one business day
  • Free 30-minute scoping call
  • Mutual NDA before any detail is shared
  • Fixed price, free retest of fixed findings