A report that finds thousands of duplicate files tells you what is wrong. It does not tell you why, and you cannot safely delete until you know.
A governance director at a conference described running a file-analysis tool across a company's repositories. It reported huge numbers of duplicates. He already knew some of the reasons. The detailed report showed several he did not expect, and each pointed to a business process, not to careless users.
Three real causes
- A paperless invoice workflow. When an invoice finished its approval and payment steps, the system saved the document back with a timestamp added to the name. The result was an identical second copy of every invoice, created automatically by a process spanning several systems.
- A “booked” button. In a property purchase process, one system held the deal documents in a SharePoint site and another system had its own site behind it. When the deal was marked booked, everything was copied from one site to the other. That made 100% duplication by design.
- Copies made after a backup scare. After one company lost months of backups, staff kept their own copies: a paper copy in a drawer, a scanned copy in the invoice system, and another in a personal shared drive. That is four copies of the same invoice.
Why deleting does not work
- You do not know which copy is the official one.
- Another system may depend on a copy being where it is.
- If the process keeps running, the duplicates come back.
A better order of work
- Run the report, then sample it. Pull a small set of duplicate groups and trace each one back to where it came from.
- Map the workflow. Find the exact step that makes the copy. Ask the process owner why it exists.
- Fix it at the source. Often a step that copies a document can link to it, or move it, instead. One copy is usually enough.
- Decide which copy is the record, and where it lives, before you delete anything.
- Then clean up the backlog, in batches, with a log of what was removed.
- Add a check so new duplicates are caught, for example a warning when someone files a document that already exists.
Why this matters for AI
AI searches whatever you give it. If ten near-identical copies exist, they all compete in the answer, and a draft or an outdated copy can win. Removing the process that creates them keeps the problem from returning. See why retention schedules decide AI accuracy and how to file business email into a governed library, which includes a duplicate check for shared threads.
Where AI can help
Once the cause is fixed, document processing can classify and tag what remains, and flag near-duplicates for a person to confirm. That is the kind of work our intelligent document processing pipelines do, with a human on decisions that count. Start with Is your data ready for AI? to see how clean your files need to be.
Illustrations in this guide are drawn from talks at an industry information-governance conference and from general practice. They are anonymized and are not Horus client results.
Part of our guide: Information governance for AI