Three documented failures, before you plan anything
1. What the chatbot says, the company says. In the case above the bot wrote that the discount could be claimed retroactively within 90 days. The company's own website, on a static page, said the opposite. The court's finding: the company has to take reasonable care that its communications are accurate and not misleading.
This was not a technical fault but a contradiction between two of the company's own sources. Keeping them in step is the subject of the knowledge base freshness article, with a monthly routine.
2. The bot invented a policy. In April 2025 the customer support bot of a developer tool — signing its emails with a human name — announced that the product could not be used on more than one machine. No such rule had ever existed. Behind it lay a genuine bug: a race condition was logging users out, and the bot made up an explanation for it.
The response from the company's co-founder: "Unfortunately this is an incorrect response from a front-line AI support bot." What they introduced afterwards: AI answers have been clearly labelled ever since.
This is the most instructive of the three, because it was not working from a bad source. The knowledge base simply had no such rule in it. The bot found no answer — and manufactured one.
3. A public-sector chatbot encouraged breaking the law. A large city's chatbot for business owners stated, among other things, that a landlord does not have to accept a housing voucher, that locking a tenant out is lawful, and that a business may go cash-free. All three run against local law.
And the most important detail: the bot was still live months after the documented errors became public. That is not a technical failure but an organisational one: there was no evaluation, no threshold, and no shutdown criterion.
The good news, with numbers — and what sits behind it
The most frequently quoted success story belongs to a large financial services firm. According to the company's own press release, "98% of adviser teams have adopted the assistant."
The detail figures, though, come from the vendor's case study: 100,000 documents in the knowledge base, 7,000 questions answerable before, and a document reach that rose from 20% to 80%.
Two things are worth adding. First: the 98% is the company's own statement, so a strong source — but it says "adopted", not "uses daily". Those are not the same. Second: the 20%→80% figure comes from the vendor's marketing material, with no methodology — nothing says how it was measured.
And the number everyone quotes
The claim that "95% of AI projects fail" comes from a 2025 report. It is worth knowing what sits behind it:
| Methodology | How many |
|---|---|
| Structured interviews | 52 |
| Executives surveyed at four conferences | 153 |
| Publicly announced initiatives reviewed | 300+ |
| Research period | January – June 2025 |
This is not peer-reviewed research, and the sample is not representative. The headline number should therefore be handled with care.
From the same report, though, comes a far more useful finding — with the same caveat, because it comes from the same sample: rollouts done with an outside partner succeeded 66% of the time, in-house builds 33%, and employee usage was roughly twice as high for the externally built tools.
The surprise: start with keyword search
This is the most contentious advice in the article, but there is a measurement behind it.
May 2026 brought a benchmark built specifically on internal company knowledge: roughly half a million documents from nine sources — chat messages, email, a task tracker, a document store, tickets, meeting minutes. Five hundred questions, in ten categories.
| Method | Correctness | Document recall |
|---|---|---|
| Keyword search | 68.8% | 68.4% |
| Embedding search | 51.4% | 46.0% |
Seventeen points of difference — in favour of keyword search.
The lesson is not that vector search is bad, but that company data is substantially different from the public benchmarks most recommendations are built on. Anyone who starts by buying a vector database may be starting from a worse place than someone who starts with keyword search — and measures afterwards how much the vector side adds. Measurements on this class of error are what the hybrid search post collects, side by side.
Permissions: the first decision, not the last
This is the most expensive and most underestimated part, and it is missing from most plans.
The problem is simple: the chatbot must not return anything the person asking would not be allowed to see. The implementation is not simple, and vendor documentation is unusually candid about its limits:
FOUR THINGS TO DECIDE ON DAY ONE
1. IT CANNOT ALWAYS BE TURNED ON LATER
On some platforms permission inheritance can
only be set at the first indexing run. After
that you have to build a new index.
2. A PERMISSION CHANGE IS NOT INSTANT
A change made in the source system only shows
up after the next synchronisation. There is a
window in which the old permission still holds.
3. INHERITED PERMISSIONS BEHAVE DIFFERENTLY
Permissions granted on the item itself are
refreshed on every run; a change INHERITED from
the folder or the site needs its own refresh.
4. A CHUNKED INDEX NEEDS ITS OWN SETTING
Filtering defined on the document does not
automatically extend to the chunks.
The second point is the dangerous one, and put as a concrete question it reads: what happens when somebody is let go? If the synchronisation window is measured in hours, then for hours they still reach through the knowledge base what the source system has already taken away from them.
And here is the liability clause spelled out by the vendor's own documentation: it is your responsibility to manage whether data leaves your organisation's compliance and geographic boundaries, and whether the right permissions are in place.
What does it cost, and how long does it take?
Let us start with the honest part: published time and cost figures backed by a methodology, for a company RAG chatbot, I could not find. Not from a vendor, not from a consultancy, not from research. What circulates online under the heading "RAG implementation cost" is guesswork.
What can be given from primary sources are the component prices — and from those you can build your own calculation.
[Own calculation] Assumption: an average document runs to five pages, roughly 3,300 tokens in English. The table below works throughout with that English token count — for Hungarian documents, multiply it by one and a half to two.
| Size | Documents | Tokens | One-off embedding |
|---|---|---|---|
| Small | 1,000 | 3.3 M | < $1 |
| Medium | 10,000 | 33 M | ~$5 |
| Large | 100,000 | 330 M | ~$50 |
Embedding is effectively free. That number always comes as a surprise.
Monthly running costs come in two items. Storage and search on a managed service run to a few dollars per gigabyte, plus one to four dollars per thousand queries. Answering, on the other hand, depends on the model, and this is where the twenty-four-fold multiplier lives: the same ten thousand questions a month costs somewhere between $26 and $635 depending on whether you pick the cheapest or the most expensive model.
The real cost is in neither of those: it is human work. Preparing the documents, mapping the permissions, assembling the test questions and doing the monthly maintenance. There is no published figure for that — but next to the items above it is orders of magnitude larger.
Will anyone actually use it?
Honest answer: there is no public, measured data. A published, methodologically sound longitudinal measurement of what share of deployed internal chatbots are still used a few months later is something I could not find. Companies do not publish usage curves for their internal tools.
What does exist is a 98% adoption figure in a press release, and a few self-reported surveys.
The practical conclusion comes precisely from that: if there is no basis for comparison, you have to measure it yourself. From day one of the rollout, log weekly active users and per-question feedback — because you cannot reconstruct it afterwards, and there is no industry number to compare yourself against.
The plan
0. DECIDE ON THE PERMISSION MODEL
This is the FIRST step. On some platforms it
cannot be switched on afterwards.
1. ONE NARROW AREA, 500-2000 DOCUMENTS
One department, one document type. The measured
successes are the narrow, well-bounded cases.
2. START WITH KEYWORD SEARCH
And MEASURE how much the vector side adds.
On company data keywords won by 17 points.
3. 100-150 TEST QUESTIONS, WITH AN EXPERT
Question, correct answer, and WHICH document.
Without it there is nothing to measure against.
4. MEASURE BEFORE ROLLOUT, TWO THINGS APART
Did it find the right document, and did it use
that document well. Not the same failure.
5. WRITE DOWN A SHUTDOWN CRITERION
Up front, in writing: the error rate at which
you stop it. The public-sector chatbot ran
wrong for months because nobody set one.
6. LABEL ANSWERS AS AI-GENERATED
Unambiguously. That is what the company with
the invented policy introduced afterwards.
7. LOG USAGE FROM DAY ONE
Weekly active users, feedback per question.
There is no industry benchmark - measure it.
8. NAME AN OWNER
Not a project lead: an owner who reviews monthly.
The fifth point is the one that follows most directly from the three documented failures. In none of the three cases was the problem that the system made a mistake — it was that nobody had written down in advance what happens then.
What you typically do not think of
Five questions worth asking at the planning stage, because every one of them is expensive later:
- Who is liable for the content if it answers wrongly? According to the court decision cited above: the company.
- What happens when somebody is let go? How wide is the permission synchronisation window?
- What if two documents contradict each other? That is what caused the first case.
- How much maintenance is there, and who does it?
- What is the threshold below which you shut it down?
Common misconceptions
- "The chatbot is just a helper, not an official statement." The court decision says it makes no difference whether the information comes from a static page or from a chatbot.
- "With a good knowledge base it will not hallucinate." In the case cited, the knowledge base had no such rule in it — the bot found no answer and invented one.
- "You need vector search for this." On company documents keyword search was 17 points better in a 2026 benchmark.
- "We will sort out permissions at the end." There are platforms where it cannot be switched on afterwards at all, and everywhere there is synchronisation lag.
- "The model and the storage are the main cost." Embedding is measured in cents, storage in dollars. The model runs $26-635 a month — and human work is an item orders of magnitude larger still.
- "If it is good, people will use it." There is no industry data on that. What there is: roughly twice as many people used the tools built with an outside partner.
What to take away
A company knowledge chatbot is not a technology project. None of the three documented failures was a model error: one was a contradiction between two of the company's own sources, the second was missing content, the third a missing shutdown criterion.
Three decisions have to be made in the first week, and all three are organisational: the permission model (because afterwards it is expensive or impossible), the shutdown criterion (because without it a faulty system keeps running), and the owner (because enforcement is only possible while the thing is live).
And one number worth keeping in mind while you plan: $812. That was the compensation in the case that set the precedent. The lesson is not the amount but that your chatbot's sentence is your sentence. The measurement details are in the RAG evaluation article, and the groundwork is laid by the RAG explainer pillar piece.