Most AI data leaks are not hacks. The AI simply answers using files that were already too widely shared, and nobody had looked.
A health system was merging several hospitals and clinics into a single Microsoft 365 environment. The request was simple: switch on AI across the whole thing. A consultant who later described the case at an industry conference said the question nobody had asked was what exactly will this AI be able to see?
An HR policy that was sensitive, misfiled, and already superseded ended up in an answer given to an employee who should never have seen it. As the speaker put it, AI does not change your security. It amplifies the mistakes that were already there.
Why AI surfaces files people forgot existed
An AI assistant cannot read every file at once. It has a limited attention span, so the system breaks your documents into small pieces, stores a searchable index of those pieces, and pulls the most relevant ones when someone asks a question.
That design creates three risks:
- Permissions may not travel with the pieces. A file had access rules. Does each piece in the index carry them, or only the original?
- Labels may not travel either. A sensitivity label or retention rule on a file means nothing if the piece in the index has lost it.
- The AI cannot tell old from new. It retrieves what looks relevant, including drafts, duplicates, and superseded policies.
The three ways AI reaches your data
- As the user. The AI uses the signed-in person's credentials, so its access is theirs. To know what it can see, audit what each person can open.
- As its own identity. You give the AI agent an account with specific permissions. This is cleaner, but many software products still charge a license for each agent identity.
- Through a connector. A connector can make AI feel like it just works, but it needs credentials. Too often an IT person uses their own broad account, and the AI inherits all of it.
What to fix first, in order
You do not need a perfect data estate before you start. You need a small, clean starting point and a plan to widen it.
- Pick one pilot repository and name a person who owns it. A team is not an owner. If nobody owns it, keep it out of AI for now.
- Measure what share of it is labeled and current. If you cannot measure it, that is your first task.
- Fix sharing before you connect. Remove company-wide links and old external shares. These spread quickly when separate systems merge.
- Clear out expired and duplicate files. Fewer competing versions means fewer wrong answers.
- Keep a person in the loop for low-confidence or sensitive answers, and keep citations so reviewers can check the source.
- Widen the scope in steps. Switch AI on everywhere except the risky sites, then add sites as they pass your checks.
If AI is already switched on
Most organizations already have some AI in use, officially or not. If yours does, reduce its reach first, then widen it piece by piece. The access an AI application holds is usually defined in one place. Narrowing that list immediately cuts off what it can read.
Narrowing access may not remove what the AI already indexed. Ask your vendor how deleted or restricted content leaves the index, and test it by asking questions the AI should no longer be able to answer.
Private AI is part of the answer
Where the AI runs matters too. Public AI tools can retain chat history and, depending on the plan, use content for training. Hosting models inside your own cloud account keeps your documents in your environment and under your permissions. Our generative AI work on Amazon Bedrock is built that way, and the law firm opinion tool shows it in a regulated setting.
Where to go next
If you want a plain checklist, start with Is your data ready for AI? To decide which AI projects are worth building at all, see our AWS AI assessment. Related guides: why retention schedules decide AI accuracy and the legal hold and e-discovery checklist.
Illustrations in this guide are drawn from talks at an industry information-governance conference and from general practice. They are anonymized and are not Horus client results.
Part of our guide: Information governance for AI