# AI red teaming: can your chatbot be talked into a data breach?

Source: https://dpdp.miravlabs.com/services/ai-red-teaming/ · Updated 2026-11-13 · DPDP Ready by Mirav Labs

**Short answer:** AI red teaming tests whether your chatbot, copilot, RAG system or agent can be tricked into leaking personal data or taking unsafe actions. Under the DPDP Act this is a security question: section 8(5) and Rule 6 require reasonable safeguards from 13 May 2027, and an AI that discloses one person's data to another can be a personal data breach that must be reported under section 8(6) and Rule 7.

## In plain words

If your business runs a chatbot, a copilot or an AI agent, someone will try to trick it. A clever message can make it reveal another customer's details, or do something it should not, like issue a refund or send an email. Under India's data protection law, that is a security failure, and a leak may have to be reported.

- **₹250 crore** maximum penalty for weak security
- **72 hours** to send the full breach report to the regulator
- **13 May 2027** when the security and breach rules apply

### What it means for you

- Your AI tool is part of your security. The law asks for reasonable protection of personal data, whatever system handles it.

- A chatbot that shows one customer another customer's order is a data leak or breach, not just a bug.

- An agent should only be able to do what the logged-in person is allowed to do.

- Instructions hidden in an email, a web page or an uploaded file can steer the AI. Testing has to cover these too.

- Testing once is not enough. A new model, prompt or tool can reopen old holes.

**Example:** An online pharmacy in Chennai adds a chat assistant that can look up orders. In a test, one fake customer account gets the assistant to read out another fake customer's address and medicines by pretending to be a support agent. The fix: the order lookup checks who is logged in, whatever the chat says.

### What to do

- List every AI tool you run, what data it can see and what it can do.

- Give written permission and agree what will be tested, using test accounts and fake data.

- Try to trick it: make it leak data and take actions it should not.

- Fix what you find, then test again.

- Turn each finding into an automatic test that runs whenever you change the model or prompts.

The law: sections 8(5) and 8(6) of the DPDP Act and Rules 6 and 7 of the DPDP Rules, 2025. Applies from 13 May 2027.

## The detailed guide

## Key takeaways

- In our reading, a chatbot or agent that discloses one person's data to someone else is a personal data breach under section 2(u), and section 8(6) has no materiality threshold.
- Rule 6 does not mention AI, but its access control, logging and “measures to make sure safeguards are observed” apply to any AI system that touches personal data.
- A useful test covers prompt injection, data leakage, agent permissions and the model's own supply chain, not just offensive answers.
- Enforce permissions in the tools and APIs an agent calls, never only in the prompt.
- Turn each finding into a regression test that runs whenever the model, prompt or tools change.

Conventional security testing assumes software follows rules. An LLM application follows instructions, and it cannot always tell your instructions apart from an attacker's. A support bot, a sales copilot, a RAG search over your documents or an agent that can call your APIs can be persuaded, through ordinary-looking text, to reveal data or take actions you never intended.

When that data is personal data, the problem stops being only a product issue. This page explains why AI security failures fall under the DPDP Act, the attack classes we test for, how an AI red teaming engagement typically runs, and what you get at the end.

## Why is an AI security failure a DPDP problem?

Section 8(5) of the Act requires every Data Fiduciary to take reasonable security safeguards to prevent a personal data breach. From 13 May 2027, [Rule 6](https://dpdp.miravlabs.com/guides/dpdp-security-safeguards/) sets the minimum, including access control over computer resources (Rule 6(1)(b)), visibility of access through logs and monitoring (Rule 6(1)(c)) and technical and organisational measures to make sure the safeguards are observed (Rule 6(1)(g)). None of this mentions AI, and none of it exempts AI. A chatbot that queries your customer database is a computer resource processing personal data like any other.

Section 2(u) defines a personal data breach to include unauthorised processing and accidental disclosure, sharing or use that compromises confidentiality, integrity or availability. In our reading, an assistant that answers one customer with another customer's details fits that definition, and so does an agent that changes a record because a crafted message told it to. Section 8(6) requires **every** breach to be intimated to the Board and each affected person, with no materiality threshold. Your legal counsel decides how a specific incident is classified; our job is to make it less likely to happen and easier to investigate if it does.

Rule 7 then asks for a detailed report to the Board within 72 hours of becoming aware, including the facts, the reasons and the remedial measures. That is hard to write if your AI system keeps no record of which prompts led to which data being retrieved. CERT-In's separate 6-hour reporting direction also continues to apply to reportable cyber incidents. See the [breach notification guide](https://dpdp.miravlabs.com/guides/dpdp-breach-notification/) for the full sequence.

Two more points matter. Section 8(1) keeps you responsible for processing done on your behalf by processors, so a hosted model or AI platform you use does not move the duty away from you. And the penalty for failing to take reasonable security safeguards is up to ₹250 crore under the Schedule, with up to ₹200 crore for failing to intimate a breach. The [penalties guide](https://dpdp.miravlabs.com/guides/dpdp-act-penalties/) explains how the Board weighs these.

## What attacks does AI red teaming test for?

We organise testing around the [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/) and the [MITRE ATLAS](https://atlas.mitre.org/) knowledge base of adversary tactics and techniques against AI systems. Both are public references, which makes findings easier for your engineers and auditors to follow. The table describes the main classes in plain words. We do not publish working attack strings.

| Attack class | In plain words | What can go wrong with personal data |
|---|---|---|
| Direct prompt injection and jailbreaks (OWASP LLM01) | A user writes messages designed to make the model ignore its instructions or safety rules, sometimes across many turns or through role-play | The assistant drops its restrictions and reveals records, internal notes or another user's details |
| Indirect prompt injection (OWASP LLM01) | Instructions are hidden inside content the AI reads: an email, a support ticket, an uploaded file, a web page, a product review | A copilot summarising an inbox or a ticket queue is steered into sending data out or acting for the attacker |
| Sensitive information disclosure (OWASP LLM02) | The model returns data it had access to but the user should not see, from context, memory or fine-tuning | Names, phone numbers, health or financial details appear in answers to the wrong person |
| System prompt leakage (OWASP LLM07) | The hidden instructions behind the assistant are extracted | Exposed instructions reveal business rules, internal endpoints or, worse, credentials placed in the prompt |
| Retrieval and vector store weaknesses (OWASP LLM08) | A RAG system fetches documents without checking whether this user may see them | One tenant's or employee's documents surface in another's answers |
| Excessive agency (OWASP LLM06) | An agent has more tools, permissions or autonomy than its task needs | It issues refunds, edits accounts, sends messages or deletes data because a message asked it to |
| Improper output handling (OWASP LLM05) | Model output is passed to a browser, database or another system without checks | Rendered links or images quietly carry data to an outside address; output becomes an injection into other systems |
| Supply chain, data and model poisoning (OWASP LLM03, LLM04) | Third-party models, plug-ins, datasets or fine-tuning data are tampered with or untrusted | A component behaves differently from what you assessed, or leaks what it is given |

Because users in India write in many languages, we test in the languages your users actually use, including mixed Hindi and English where relevant. A filter that has only been tried in English has not really been tested.

## How does an AI red teaming engagement run?

Every engagement starts with written authorisation from someone who can give it, and an agreed scope. We do not test systems, accounts or third-party platforms outside that scope, and testing respects the terms of any hosted model or platform you use, which we check with you during scoping. A typical engagement follows five steps.

### 1. Scope: models, agents, tools and data

We map the system as it really runs: which models are used and where they are hosted, the system prompts, the retrieval sources, every tool or function an agent can call, the identity each tool runs as, and what personal data each part can reach. We agree rules of engagement: environments, test accounts, synthetic data, testing windows, stop conditions and named contacts on both sides. The output is a short threat model that links each component to the attack classes above.

### 2. Attack: prompt injection and jailbreaks

We try to make the system break its own instructions, directly in the chat and indirectly through the documents, emails, pages and records it reads. We test single messages and long conversations, different personas, different languages and different entry points (web chat, app, messaging channels, APIs). The goal is to learn which guardrails hold and which only hold for polite users.

### 3. Leak test: personal data and secrets

We seed the test environment with synthetic people carrying marked values, then try to get one test user's session to reveal another's. We try to extract the system prompt, credentials, connection details and internal notes. For RAG systems, we check whether retrieval respects each user's access rights. We also look at where transcripts end up: logs, analytics and evaluation tools can hold more personal data than the product itself, and Rules 6(1)(e) and 8(3) mean logs are kept for at least a year, so they need to be minimised and protected.

### 4. Permission test: can the agent take unsafe actions?

For each tool, we ask what the worst action is that the agent can be talked into. We test whether authorisation is enforced by the tool or API itself, based on the real user, or only by instructions in the prompt. We check high-impact actions (payments, refunds, account changes, outbound messages, deletion) for confirmation steps or human approval, and we check limits on volume and cost. This is where AI testing meets [API security](https://dpdp.miravlabs.com/services/api-security/): an agent is only as safe as the APIs it is allowed to call.

### 5. Fix and retest: findings, fixes and a regression suite

We report findings as we go for anything serious, then in a full report ranked by severity and impact on personal data. Your engineers fix; we can pair with them on the harder fixes. We retest each fix, then turn every confirmed finding into an automated test case your team can run whenever the model, prompt, retrieval sources or tools change. Model updates and prompt edits can quietly undo earlier fixes, so the regression suite is one of the most lasting deliverables.

## What do you get at the end?

| Deliverable | What it contains | Who uses it |
|---|---|---|
| Scope and rules of engagement | Signed authorisation, systems in and out of scope, environments, test data, stop conditions, contacts | Security lead, legal counsel, the testing team |
| AI system map and threat model | Models, prompts, retrieval sources, tools, identities and personal data flows, linked to OWASP and MITRE ATLAS categories | Engineering, architecture, privacy team |
| Findings report | Each finding with severity, conditions to reproduce (shared securely), personal data at risk and a recommended fix | Engineering and product owners |
| DPDP safeguard mapping | Which Rule 6 measure each finding and fix relates to, for your security evidence pack | DPO or privacy lead, internal audit |
| Executive summary | Plain-language risk picture and the decisions needed, without technical detail | Leadership and the board |
| Regression test suite | Automated cases for every confirmed finding, ready to run in your build pipeline | Engineering and QA |
| Retest report | Status of each finding after fixes: closed, partly fixed or open with a plan | Security lead, auditors, enterprise customers' reviewers |

The mapping and retest report are evidence that you tested and fixed, which supports your Rule 6 file. They are not a certificate, and no test can show that a system has no remaining weaknesses.

## How is AI red teaming different from a normal penetration test?

A conventional penetration test

- Networks, web apps, mobile apps and APIs

- Inputs follow fixed formats and rules

- The same input usually gives the same result

- Findings are mostly code or configuration flaws
AI red teaming adds

- Natural-language instructions that can be overridden

- Answers that vary from run to run, so tests are repeated

- Content the AI reads as an attack path

- What an agent can be persuaded to do with its tools
You usually need both. AI features usually sit on ordinary APIs, databases and identity systems, and a weakness there is easier to exploit through an AI front end. Where a conventional test is already planned, we can coordinate so the two cover each other rather than overlap.

## Which controls usually fix AI security findings?

Fixes depend on the system, but the controls we recommend most often are straightforward engineering:

- **Permissions in code, not prompts.** Every tool and retrieval call checks the real user's rights. The prompt can describe the rules; it cannot be the only thing enforcing them.

- **Least privilege for agents.** Give each agent only the tools and data its task needs, with separate credentials and scopes.

- **Human approval for high-impact actions.** Payments, deletions, account changes and outbound messages wait for a confirmation the model cannot give itself.

- **Treat retrieved content as untrusted.** Documents, emails and web pages are data, never instructions, and output is checked before it reaches a browser or another system.

- **No secrets in prompts.** Credentials live in a secret store and are used by the tool, not shown to the model.

- **Minimised, protected logs.** Keep the trail Rule 6 asks for, with identifiers masked where you can, so you can investigate and report without creating a second leak.

Some issues are governance decisions rather than bugs: which data an AI feature should see at all, and who signs off on new tools. Our [AI governance](https://dpdp.miravlabs.com/services/ai-governance/) and [data readiness for AI](https://dpdp.miravlabs.com/services/enterprise-data-readiness-for-ai/) services cover that side. If you are ever notified as a [Significant Data Fiduciary](https://dpdp.miravlabs.com/guides/significant-data-fiduciary/) (none has been notified yet), Rule 13 adds due diligence that technical measures, including algorithmic software, do not risk the rights of Data Principals.

## How to start

- **List your AI features.** Chatbots, copilots, RAG search, agents, and internal tools staff use with customer data.

- **Pick the riskiest first.** Usually the one that can see the most personal data or take the most consequential actions.

- **Prepare a test environment.** Test accounts, synthetic records and the same prompts, tools and retrieval as production.

- **Agree scope and authorisation.** In writing, with your security lead and, where needed, your legal counsel.

- **Test, fix, retest, automate.** Then rerun the regression suite on every model or prompt change, and add the evidence to your [DPDP compliance checklist](https://dpdp.miravlabs.com/guides/dpdp-compliance-checklist/) work.

AI red teaming is one of our adjacent engineering [services](https://dpdp.miravlabs.com/services/), alongside DPDP implementation. To scope a test of your chatbot, copilot or agent, [get in touch](https://dpdp.miravlabs.com/#contact) and we will usually start with a short call to map the system and agree what should be tested.

## Frequently asked questions

### What is AI red teaming?

AI red teaming is authorised adversarial testing of an AI system: testers try to make a chatbot, copilot, RAG system or agent break its own rules, reveal data it should not, or take actions it should not. The aim is to find those weaknesses in a controlled test, fix them, and keep checking that the fixes hold.

### What is prompt injection testing?

Prompt injection testing checks whether text from a user, or from a document, email or web page the AI reads, can override the instructions you gave the model. OWASP lists prompt injection first in its Top 10 for LLM Applications 2025. Testing covers direct attempts in the chat and indirect ones hidden in content the system retrieves.

### Is a chatbot data leak a personal data breach under DPDP?

In our reading, yes, if personal data is disclosed to someone not authorised to see it. Section 2(u) defines a personal data breach to include unauthorised processing and accidental disclosure, and section 8(6) requires every breach to be intimated to the Data Protection Board and each affected person, with a detailed report to the Board within 72 hours under Rule 7. Confirm the position for a specific incident with your legal counsel.

### Does the DPDP Act require AI red teaming?

The Act and Rules do not name AI red teaming. Section 8(5) requires reasonable security safeguards, and Rule 6 requires technical and organisational measures to make sure those safeguards are observed. Testing an AI system that handles personal data is one practical way to check and evidence that, but it is recommended practice rather than a named legal requirement.

### How is AI penetration testing different from a normal pen test?

A normal penetration test looks at networks, applications and APIs, where inputs follow fixed rules. An AI system takes instructions in natural language and can be persuaded, so testing adds prompt injection, jailbreaks, leakage through answers and retrieval, and checks on what an agent can be talked into doing. The two complement each other, because AI tools usually call ordinary APIs.

### Do you test AI systems with real customer data?

We aim not to. We prefer a test or staging environment seeded with synthetic records and test accounts, so a successful leak exposes only fake data. If testing must touch production, the scope, data handling and stop conditions are agreed in writing first, and any real personal data we come across is reported to your named contact rather than kept.