By Scott Sumner | AI at Work | Published October 12, 2026

An office librarian separates a current purple reference tray from a box of historical papers, illustrating the difference between active AI guidance and an archive.

An AI knowledge base should save your team from hunting through old files. That benefit disappears when yesterday’s policy is still available as today’s answer.

Keep an AI knowledge base current by separating approved guidance from historical records, recording when each rule applies, and checking the search results after updates. Uploading a replacement document does not by itself prove the old answer has stopped appearing.

This is not a reason to abandon document search. It is a reason to treat the collection as a working reference, rather than a storage cupboard with a chat box attached.

Jump ahead

Consider a fictional agency with two onboarding guides. The older guide says every project includes two training sessions. The current offer includes one session, with additional training available separately.

Someone asks the internal assistant, “What training do we include for a new customer?” It finds the older guide and quotes the two-session line correctly. It even supplies a link.

Nothing in that sentence needs to be invented for the answer to cause trouble. The source exists. The quotation is faithful. The wrong version has authority it should no longer have.

Now add an existing customer whose original agreement really does include two sessions. Deleting every mention of the older offer would create a different mistake.

The business needs to distinguish a current default from an applicable historical agreement. “Use the newest file” is too crude. A recently uploaded scan of an old agreement is not a new policy.

My preference is to settle that distinction before connecting a whole shared drive. More files can mean more useful coverage, but only if the assistant has a dependable way to select what applies.

A common approach is retrieval-augmented generation, often shortened to RAG. In plain English, the system searches a collection and supplies relevant passages to the model before it writes an answer.

OpenAI’s retrieval documentation describes semantic search as finding material with similar meaning, including passages that share few or no keywords with the question. When files enter its vector stores, they are split into chunks, converted into searchable representations and indexed.

That is useful. A team member should not need to remember the exact wording in an operations manual to find the answer.

But a relevance match is not a business approval. Two conflicting instructions can both be highly relevant to the same question. A citation tells you where a claim came from; you still need to establish whether that source governs this situation.

That distinction is my analysis of the workflow, not a claim that every retrieval product handles conflicts identically. Some products offer source controls, filters and ranking options. What matters is how your actual implementation uses them.

Microsoft’s Copilot Studio documentation makes another useful distinction. Turning off “Allow ungrounded responses” does not guarantee that an agent never uses general knowledge. Microsoft says the setting blocks turns that used no knowledge source or tool at all; a model may still combine general knowledge with retrieved information.

So “it only answers from our files” deserves a settings check, not a nod. Even a tightly restricted collection still needs accurate, applicable content.

Build a current reference, not a second archive

Start by choosing the job. An assistant answering routine questions about your current service should not automatically search every historical proposal, brainstorm and abandoned draft.

For that job, maintain a small approved reference set. Keep historical records according to your existing business requirements, but separate them from the default answer path. Archiving here means controlling retrieval, not destroying records.

A useful source register can be a plain table:

Field What it resolves
Document identity and version Which source actually supports the answer?
Status Is it approved, a draft, or superseded?
Effective period When does the rule apply, rather than when was the file uploaded?
Scope Which service, region or customer group does it cover?
Replacement source Where should a current question go instead?
Responsible reviewer Who can resolve an unresolved conflict?

These are suggested operating fields, not a universal software specification. A small team can start with fewer fields if it can still answer the questions.

Where the software supports it, use those fields to constrain retrieval rather than merely asking the model to ignore obsolete information. OpenAI’s documented attribute filters can narrow a search before semantic retrieval, including by date or other file attributes.

The filter is only as good as the metadata and application logic. Writing “approved” in a spreadsheet will not protect a chatbot that never reads that field. Have the implementer show which sources an ordinary current-policy question can actually reach.

Access is a separate question too. A document can be current and still inappropriate for a particular user. Do not expand access to private customer records just to improve answer coverage.

Make replacement a complete operation

When guidance changes, the work is not finished at upload. A useful replacement procedure includes:

  1. Approve the new wording and identify what it replaces.
  2. Update the reference set and its status, scope and effective dates.
  3. Exclude superseded material from default current-answer retrieval, while preserving any needed historical route.
  4. Confirm the system finished processing the change.
  5. Ask the questions affected by the change and inspect the retrieved sources.

The processing step is easy to miss. OpenAI documents that some vector-store file operations are asynchronous. It also says removing a file from a vector store is eventually consistent: search results may briefly include content from a removed file.

That is a specific documented behavior, not a claim about every knowledge platform or a promised waiting period. The practical lesson is to check the system’s status and then its behavior. Do not tell staff “the answer is fixed” solely because someone clicked Remove.

If the update changes a consequential promise, route that question to an approved manual answer until the revised search path has been checked. The temporary inconvenience is preferable to distributing the old promise at speed.

Test which version the assistant uses

For the fictional training example, I would test four questions:

  • How many training sessions come with a new project?
  • Does the old two-session offer still apply to new customers?
  • What should we do if an existing customer’s agreement includes two sessions?
  • Which approved document supports the answer?

The first checks the normal case. The second tempts the assistant with outdated wording. The third checks that historical exceptions have not been erased. The fourth makes the evidence inspectable.

For an account-specific question without access to the actual agreement, a useful answer should ask for the approved record or route the question to someone who can check it. It should not convert the current default into a claim about every customer.

Check retrieved passages separately from the final response when the tool allows it. If the correct document was never retrieved, rewriting the answer prompt may not address the cause. If the right passages were retrieved but misread, the failure is different.

Keep these questions with the workflow so you can run them after future changes. The guide to AI regression testing covers that wider practice. Here, the specific target is document authority and applicability, not merely whether the new wording sounds better.

Start with one repeated question

You do not need to reorganize the entire business archive this week. Choose one question staff ask often, such as what onboarding includes. Find the current answer, the superseded answer and any real exceptions.

Then make the assistant’s search boundary match that decision. This is a small enough task to inspect and a useful place to discover how updates actually propagate.

The same approved facts should inform your public service explanation, without exposing internal or customer-specific material. If the website promises something different from the current offer, an internal assistant cannot repair the buyer’s expectations. That connection belongs in a broader AI Findability review.

For help connecting the public explanation with the real service, talk to Sumner.ai about your website and AI workflow. Start with the question people keep asking, not the number of documents you can upload.

Sources

Documentation checked October 12, 2026. The agency example is fictional. The source register and operating procedure are Sumner.ai recommendations, not reported client results.