By Michael Walker, CEO, Horus Technology
An AI assistant is only as safe as the label on each piece of text it retrieves. On AWS, you keep it safe by labeling documents before they are indexed, filtering every answer by who is asking, logging what was used, and testing that retired content is gone.
This guide is the build side of how AI exposes data nobody labeled. That guide covers what to fix first. This one covers how to keep the fix in place once the AI is running.
Why does AI show people files they should not see?
The index an AI searches is built from small pieces of your files, and a piece that loses its label and permissions can be shown to anyone who asks, as this guide explains.
What does a safe setup do, in order?
A safe setup does four things: it labels each document before indexing, filters every answer by who is asking, logs what the AI used, and removes retired content on a schedule. Each one closes a gap the others leave open.
- Label. Decide what each document is and who may see it, before it is indexed.
- Filter. Apply the label as a condition on every question, based on the person asking.
- Log. Keep a record of who asked, what was retrieved and what was answered.
- Retire. Remove expired content, then test that the AI no longer answers from it.
How do you label documents so the labels reach the index?
Classify each document first, then write its sensitivity, owner, retention category and date to a metadata record that travels with it into the index. Send anything the classifier is unsure about to a person.
In Amazon Bedrock Knowledge Bases with documents in S3, each document can have a companion metadata file named after it, such as policy.pdf.metadata.json. Your classification step is what fills that file in. Our document-processing pipelines classify and extract, then route low-confidence results to a reviewer. To know how far to trust the classifier, measure it, as described in how many documents to review to trust AI classification.
How do you stop the AI retrieving what a person cannot see?
Attach a metadata filter to every retrieval request, built from who is asking. Knowledge Bases support filters such as equals, not-equals and in, which you can combine with and and or.
The filter is part of the query, so your application has to set it from the signed-in person. The knowledge base does not work out who the user is for you. Check two details before you choose a design. Some operators, such as in and not-in, are best supported on particular vector stores, and AWS lists which. And a filter can only use labels that exist, which is why labeling comes first.
You can add guardrails with denied topics and content filters on top. Treat them as a second layer, not a replacement for permissions.
How do you check what the AI actually used?
Turn on model invocation logging and keep the results. When it is enabled, each log entry records the identity that made the request, the input and the output, and it can be delivered to CloudWatch Logs or S3 in your own account and Region.
- Logging is off by default, so someone has to turn it on.
- The logs contain what people asked and what the AI answered, so protect them like any other sensitive record, and decide how long to keep them.
- Knowledge Base answers can include citations to the source pieces, which lets a reviewer check where an answer came from.
How do you make sure retired content is really gone?
Deleting a file does not prove the AI has forgotten it. Remove the content, refresh the index, then ask the AI the questions it should no longer be able to answer.
Whether a deleted file leaves the index depends on how your index is built and refreshed, so check it for your setup instead of assuming. Keep a short set of test questions and run it after every cleanup and every system update. The method is in why retention schedules decide AI accuracy.
Does the AI stay private on AWS?
It can. You can run the model inside your own AWS account, and AWS states that content sent to Amazon Bedrock is not used to improve the base models and is not shared with model providers.
For network privacy, interface endpoints let you reach Bedrock without an internet gateway, NAT device, VPN connection or Direct Connect. Our law firm proof of concept uses a private endpoint and cites a source for every asserted fact.
What can AWS not fix for you?
AWS cannot change who has access inside Microsoft 365, choose your labels, or name the person who owns a SharePoint site or shared drive. Those are your first jobs, and they come before any AWS setting.
- Sharing links. Company-wide links live in the system where the files are kept. Remove them there, especially after two systems merge.
- Owners. Name a person, not a team. If nobody owns a site or drive, keep it out of AI.
- Label rules. Decide what counts as confidential before a classifier applies it.
Where should you start?
Start with one SharePoint site or shared drive, one named owner and one test account that should see almost nothing, then widen only when that account cannot pull anything sensitive; the order of fixes is in our exposure guide.
If you want a second pair of eyes, we can review the places your files live against five checks and tell you what to fix first. Start with Is your data ready for AI?, or see how we decide which AI projects are worth building in our AWS AI assessment.
Part of our guide: Information governance for AI