Keeping the Knowledge Base Fresh

The measurement that settles how urgent this is

The weight of the benchmark comes from its method: monthly snapshots of Wikipedia were compared across half a year, and factual changes were extracted automatically, at 96.8% extraction accuracy. That produced 96,124 question-answer pairs and 219,463 documents.

The result in the presence of stale documents:

The study reports a drop of at least 20% — the table shows the absolute figures measured with stale documents present — and, at the bottom of the field, performance below the level of chance.

The key sentence, though, is this one: models stay appropriately uncertain when they find nothing, but "become prone to confident but incorrect answers when they encounter outdated information."

That turns the usual thinking around. An incomplete knowledge base is annoying: the bot says it does not know. A stale knowledge base is dangerous: the bot confidently says something wrong, and the user gets no signal that anything is off.

Why does the model not notice?

Because that is not its job. The founding principle of RAG is to trust the context — the whole reason you build it is so the model answers from your documents rather than from its own memory.

There is a 2026 measurement that puts a number on what this costs. The study looked at 4,500 records to see what happens when the retrieved text contradicts what the model otherwise knows:

And the most dramatic detail from the same source: when a document containing a widespread misconception was placed in the knowledge base, the accuracy of conventional RAG fell to 15%.

In plain terms: if you have an old, withdrawn document in your knowledge base, the system will hand it back most of the time — not because the model is bad, but because that is exactly what you asked it to do.

(The authors of the study warn about this themselves: the mechanism of the improvement differs from one model family to another, so there is no single established fix for conflicts.)

Which points straight at where this has to be solved: not at the model, but at indexing. Once the wrong document is among the results, it is already too late.

The four operations that do not happen by themselves

Keeping things fresh consists of four operations, and two of them usually do not work on their own.

Deletion is the most dangerous, because it is silent. You take the document out of the folder, the system keeps working, and the chatbot hands back for months something that no longer exists.

Vendor documentation shows this varies by platform: on some it works automatically; on some only if you set it up at the first indexing run; on some only in full reconciliation mode; and on some you have to switch on a separate cleanup setting.

The practical task fits in one sentence: find the word "deletion" in your system's documentation before you go live.

Reindexing is cheaper than you think

[Own calculation] This one usually comes as a surprise:

A monthly full reindex of ten thousand documents costs sixty dollars a year. This is not a cost question — it is an operations question.

The real cost sits elsewhere: in rebuilding the index. Vendor documentation lists precisely what can be changed while running and what forces you to throw the whole thing away and rebuild:

CAN BE DONE ON THE FLY
- adding a new field
- changing a scoring profile, synonym list or
  semantic configuration
- toggling whether an existing field is returned

REQUIRES A FULL REBUILD
- deleting a field
- CHANGING A FIELD DEFINITION (type, searchable,
  filterable, sortable)
- assigning or changing an analyzer
- switching service or pricing tier

The second line of the second list is what surprises everyone: swapping the embedding model is a field-definition change. If you move to another model, or the number of dimensions changes, you need a full index rebuild. The cost is not the fifty dollars — it is the downtime and the risk.

There is a technical answer to this, and it is the difference between "maintenance downtime" and "the user notices nothing": the index alias. Your application refers to an alias rather than to a concrete index; you build the new index alongside, test it, and when it is ready you point the alias at it.

The order, according to the documentation:

1. Save the current index definition
2. If you have spare capacity, keep the old one
   while you build and test the new one
3. Build the new one with the changed fields
4. Load it from the external source
5. Flip the alias over

(For the time a reindex takes I found no published figure — it depends on API limits, chunk size and the write speed of the store. Measure it once on a thousand of your own documents and scale it up.)

Contradictory documents

This is the hardest part, because it is not a technical problem. Two documents say different things about the same subject — the old policy and the new one, the Hungarian version and the English one, the contract template and the actual contract.

The model will not solve it for you. The measurement above shows that handling conflicts gives different results from one model family to another, and there is no single established fix.

The only approach that works is on the indexing side, in three steps:

1. Mandatory validity metadata on every document. Valid from, valid until, version, status. Without it, no filtering is possible at all.
2. Take the withdrawn document out of the index, do not filter it. Filtering can be forgotten, switched off, and on chunked indexes it does not always extend to the chunks. What is not in there cannot be handed back.
3. If the system does find a contradiction, it should not choose. The correct behaviour is to flag the conflict and show both sources — not to pick one of them.

The third point matters because it is the difference between a useful and a dangerous system. A chatbot that says "I have two contradictory sources on this, take a look at them" is more useful than one that confidently quotes one of them.

Who owns it?

Honest answer: a targeted industry survey of who owns the knowledge base after an AI rollout is something I could not find. In our experience, though, this role is simply not assigned in most organisations — and that is exactly why knowledge bases go stale.

What can be recommended responsibly: the owner should be whoever owns the source documents, not whoever runs the system. The HR policy is owned by HR, not by IT. Once those come apart, updating is always the other department's job.

The routine

1. A VALIDITY FIELD ON EVERY DOCUMENT
   Valid from, valid until, version, status.
   Without this one, nothing else works at all.

2. CHECK DELETION FIRST
   Put in a test document, delete it, then ask
   about it. If it comes back, you have work to do.

3. WEEKLY INCREMENTAL, MONTHLY FULL
   The incremental run is fast and cheap. The full
   monthly recompute catches what it got wrong.

4. AN INDEX ALIAS FROM DAY ONE
   Point the application at an alias. Without it,
   every structural change means downtime.

5. FLAG THE CONFLICT, DO NOT RESOLVE IT
   Two sources, two dates, the user decides.
   More useful than one confident mistake.

6. RERUN TWENTY QUESTIONS EVERY MONTH
   The same twenty questions, month after month.
   If it degrades, you know before a complaint.

7. NAME AN OWNER PER DOCUMENT TYPE
   Whoever owns the source, not whoever runs the
   system.

8. REVIEW YEARLY WHAT IS NO LONGER NEEDED
   A bigger knowledge base is not the goal. Accuracy is.

The second point takes five minutes and catches the most common silent failure. It is worth doing before you go live.

Common misconceptions

• "An incomplete knowledge base is the bigger problem." The other way round. The measurement says that on an empty result the model stays uncertain; on a stale result it is confidently wrong.
• "If I delete the file, it leaves the system." On several platforms it does not. Propagating deletion is a separate setting, and on some it can only be switched on at the first indexing run.
• "The model will notice the data is old." It will not. The founding principle of RAG is to trust the context — that is exactly why you built it.
• "Reindexing is expensive." A full recompute of ten thousand documents costs five dollars. The expensive part is the structural rebuild of the index and the downtime.
• "Swapping the embedding model is just configuration." It is a field-definition change, so a full index rebuild.
• "The system should resolve the contradiction." There is no established fix for that. The correct behaviour is flagging, not choosing.

What to take away

A knowledge base does not go bad because it is incomplete; it goes bad because it is old — and in the measurement that difference is at least twenty percent.

Three things prevent it, and all three can be set up in the first week: validity metadata on every document, a check that deletion propagates, using a test file, and an index alias, so that structural change does not mean downtime.

And one monthly routine that costs five minutes: run the same twenty questions again. Not to get a good number, but to notice when it starts to degrade — because users will not tell you. They will just stop using it. Planning the rollout is what the company knowledge chatbot article lays out, step by step, and the measuring side belongs to the RAG evaluation post, with the numbers.