When people hear the phrase data leak, they often imagine an attacker breaking into a server, stealing credentials or exploiting a software flaw.

AI introduces another possibility: sensitive information can be exposed even when nobody has broken into anything.

This article is mainly about your own users, connectors and AI outputs moving information beyond the audience you intended. It is not about a model mysteriously revealing another customer's private training data.

No firewall necessarily failed. The system may simply have been allowed to see more than the task required, or an output may have carried private context somewhere it should not have gone.

Exposure is not always a traditional breach

A conventional data breach usually involves unauthorised access.

AI exposure can happen through authorised access operating exactly as configured.

An employee may paste confidential material into an approved AI service. A connected assistant may retrieve more files than expected. A generated summary may include private details and then be forwarded to a client or copied into Slack.

The useful question is therefore not only "Was the system hacked?" but also "Where was this information allowed to travel?"

Four common ways information can escape its intended boundary

1. Sensitive information is placed directly into the prompt

This is the simplest route.

Someone pastes a customer list, internal email, contract, source code, HR note or financial information into an AI tool.

That does not automatically mean the information will be used to train a model. Providers and account tiers differ. But the information has now entered another service and may be subject to that service's retention, logging, review and workspace-access rules.

In practical terms, the AI's working context is now another place where a copy of that information exists.

Before submitting sensitive material, ask who may be able to access that copy: the user, workspace administrators, the provider under its terms, connected tools, or anyone who later receives the generated answer.

2. A connected AI retrieves more than the task needs

Modern assistants can search email, cloud storage, CRM systems and internal knowledge bases.

Imagine asking an AI to summarise project updates for a client. If its permissions are too broad, the search may also retrieve salary information, legal drafts or internal strategy documents that happen to mention the same client or project name.

OWASP's LLM06:2025 Excessive Agency guidance highlights the danger of giving AI systems more functionality or permissions than their task requires.

The problem is not that connectors are inherently unsafe. The problem is allowing a connector to see a much larger information set than the job actually needs.

3. Private context appears in an output that travels further

An AI can only work with the context available to it, but once private material is in that context it can influence the answer.

Suppose an assistant uses both public project files and internal cost information to prepare a client update. The draft may include a margin, internal comment or confidential code snippet that was never intended for the client.

That becomes an exposure when someone forwards the answer, pastes it into email or Slack, copies it into a support ticket, or publishes it without checking.

OWASP's LLM02:2025 Sensitive Information Disclosure guidance covers the risk of sensitive information being revealed through model outputs and application context.

The model does not automatically know which business facts are contractually confidential simply because they are confidential to your organisation.

4. Permissions break down in shared systems

Enterprise AI search and connectors should normally respect the permissions of the underlying system.

But if a shared index, connector or workspace is configured too broadly, or source permissions are not carried through correctly, an AI can surface material to someone who should not be able to see it.

For example, an internal search assistant might return text from a colleague's private draft because the indexed copy was given wider access than the original document.

That is not an inevitable property of AI search. It is a permissions failure.

Two everyday business examples

Customer support summarisation - prompt and output exposure. A support ticket contains payment details and a home address. An employee sends the full ticket to an AI service even though only the complaint itself was needed. The resulting summary is then pasted into another system. No attacker was involved, but unnecessary sensitive information crossed several boundaries.

Cross-department retrieval - over-retrieval and weak permissions. A marketing employee asks an internal AI for background on a product launch. The assistant retrieves an unreleased legal draft using the same project codename because the connected search index is broader than the employee's intended scope.

These are both examples of a system doing something it was technically allowed to do while producing a result the organisation did not actually want.

Security controls and provider privacy policies are different things

It helps to separate two questions.

Provider privacy controls cover issues such as retention, model improvement, human review and the contractual handling of your data.

Your security controls determine what users and connected AI systems can access in the first place.

A provider can make a strong commitment not to train its models on your business data and you can still have a poorly scoped connector that retrieves confidential files.

Those are different risks.

How to reduce the risk

The useful controls are familiar ones:

  • Minimise the data. Give the AI only the information the task needs.
  • Limit connector permissions. Avoid broad inbox, drive or database access where narrower access is available.
  • Preserve source permissions. Make sure AI search respects the same user-level access controls as the systems it retrieves from.
  • Review outward-facing outputs. Check client emails, reports, code and published material before sending them beyond the original audience.
  • Think about the next copy. If an AI answer is pasted into another chat, email, ticketing system or collaboration tool, the information has crossed another boundary.

This is the data-exposure side of the same least-privilege principle discussed in Too much power: why AI agents shouldn't have access to everything.

Key takeaways

  • AI can expose sensitive information without a traditional hack when authorised users, connectors or outputs move data further than intended.
  • The main controls are data minimisation, least privilege, preserved source permissions and careful review of outward-facing outputs.
  • "Not used for model training" is an important privacy commitment, but it does not by itself prevent over-retrieval, over-sharing or poorly scoped integrations.

A note on changing products

AI connectors, workspace permissions and provider controls change quickly. securedby.ai reviewed the sources above on 14 September 2026. Recheck current product and standards documentation before making a high-stakes security decision.