Loading…
Loading…
Enterprise search with RAG: why keyword search fails, how retrieval actually works, the permissions problem nobody writes about, how to evaluate retrieval quality, and why these projects fail.
The model is the commodity and retrieval is the product. Four of the five stages in a RAG enterprise search system are retrieval engineering, and only one is the model, yet project effort is usually allocated in the opposite proportion.
Keyword search failed on vocabulary mismatch rather than on speed, but it still wins on exact identifiers such as invoice numbers and error codes, so a serious system keeps both.
Permissions are the hardest part of enterprise search and have no elegant solution. A language model has no concept of authorisation and will summarise whatever retrieval hands it.
Build a real evaluation set of fifty to two hundred questions with known answers, and measure retrieval separately from generation, because they are different failures with different fixes.
Fix the corpus first. A system retrieving confidently from a superseded document is working correctly and answering wrongly.
Search for enterprise search and every commercial result is a company selling an enterprise search product. Elastic, Glean, Coveo, Sinequa, Moveworks, IBM, Hyland, Stravito. They are mostly good products and the material they publish is mostly accurate.
The most interesting result on that page is not any of them. It is a Reddit thread ranking in the top ten, where someone at a five hundred person company describes information retrieval as a daily scavenger hunt across Slack, Google Drive, Notion, Jira and buried email threads. Google is ranking that thread because it is the only result on the page written by someone with the problem rather than someone selling the solution.
This guide is written from the build side. We are an engineering firm, so we are usually the people who implement these systems and occasionally the people who have to explain why the impressive demo does not survive contact with a real corpus.
Enterprise search is the ability to find information held inside your organisation, across the many systems that hold it, from one place. The definition has not changed in twenty years. What changed is what finding means.
Classical enterprise search returns documents. You give it words, it gives you a ranked list of files containing those words, and you open them and read until you find the answer. This is lexical or keyword search, and the dominant implementations are built on inverted indexes.
RAG based enterprise search returns answers. You ask a question in your own words, the system retrieves the passages most likely to contain the answer, and a language model composes a response from those passages, ideally with citations back to the sources.
The difference that matters commercially is that the first puts the synthesis work on the employee and the second does it for them. The difference that matters technically is that the second can be confidently wrong in a way the first cannot, which is why the engineering around it is more demanding rather than less.
It is worth being precise about this, because the failure modes explain what RAG has to fix.
Vocabulary mismatch is the first. The searcher and the document use different words for the same thing: someone searches for time off policy, the document says annual leave entitlement, and nothing matches. Every organisation has an internal dialect, and keyword search requires the searcher to already know it.
The second is that a keyword index has no notion of a question. How many days notice do I need to give before leave is a question, but the index treats it as a bag of words, most of which are noise, and matches on days, notice and leave independently.
The third is ranking without context. Classical relevance ranking weighs how rare and how frequent your terms are. It has no idea that a policy document supersedes a three year old email thread that happens to use the same words more often. And the fourth is fragmentation: the answer lives in one system, the exception lives in a chat thread, and the current version lives in someone's document, so even a perfect index of one system answers nothing.
Where keyword search still wins, and this gets forgotten in the enthusiasm, is exact identifiers. Invoice numbers, part codes, error strings, ticket references. If a user types an exact token they want exact matching, and a purely semantic system will helpfully return things conceptually similar to that invoice number, which is useless. Serious systems keep both, and that is what hybrid search means.
Five stages, and the interesting failures are concentrated in the first two rather than the last.
Ingestion and connectors pull content from the systems that hold it: document stores, wikis, ticketing systems, chat, email, databases. This is unglamorous integration work and it is usually the largest single line in the project, because every connector has its own authentication, rate limits, pagination and permission model.
Chunking splits documents into passages small enough to retrieve precisely and large enough to make sense alone. This sounds trivial and is not. Split too small and you retrieve a fragment that has lost the context that made it meaningful. Split too large and the passage contains so much unrelated material that the relevant part is diluted. Chunking that respects document structure, sections, headings and table boundaries consistently outperforms splitting on a fixed character count, and the difference in final answer quality is larger than most model upgrades deliver.
Embedding and indexing converts each chunk into a vector representing its meaning, stored in a vector index. Semantically similar text produces nearby vectors, which is what allows time off policy to match annual leave entitlement without anyone writing a synonym list.
Retrieval embeds the question the same way and fetches the nearest chunks. In a serious system this is hybrid: semantic search for meaning, keyword search for exact tokens, results combined and then reranked by a model that scores each candidate against the question directly. Reranking is one of the highest return additions available and is frequently skipped.
Generation sends the retrieved passages plus the question to a language model, which composes an answer grounded in them and cites which passage each claim came from. Notice the ratio: four of the five stages are retrieval engineering and one is the model, while project effort is usually allocated in the opposite proportion.
This is the hardest part of enterprise search, it has no elegant solution, and it is the reason many pilots never reach production.
In an enterprise, not everyone may see everything. Salary data, disciplinary records, unannounced results, legal advice, one team's material that another team should not read. Your source systems already enforce this. The search layer must reproduce it exactly, and it is a new system that has just read everything.
The failure is severe and quiet. A model that has retrieved a passage will happily summarise it for whoever asked. There is no permission check inside the generation step. If retrieval returned something the user should not see, the system will paraphrase a confidential document to an unauthorised employee in fluent prose, with a citation, and nobody will notice until it matters.
There are two broad approaches. Filtering at query time stores each chunk with the access control list of its source and constrains retrieval to what this user may see. This stays correct as permissions change and is the right default, at the cost of an authorisation check on every query and an index that must faithfully mirror permission models differing across every connected system. Separate indexes per security boundary is simpler and faster, viable when boundaries are few and stable, and degrades badly as the number of distinct permission combinations grows.
Three details are learned the hard way. Permissions change and your index must notice, because if someone leaves a project and the index refreshes weekly, there is a week in which they can still retrieve that material. The chunk inherits the document's access list rather than the folder's, because documents get moved and shared individually. And you should log every retrieval rather than only every query, because when someone asks whether the system exposed something it should not have, the query log alone cannot answer it.
If a vendor cannot describe their permission model in detail, that is the single most important thing you have learned in the meeting.
The most common cause of a disappointing system is not the model. It is that the right passage was never retrieved, so the model was asked to answer from material that did not contain the answer. It then either says it does not know, which users read as broken, or it produces something plausible from adjacent context, which is worse.
Fix the corpus first. If your document store contains four versions of the expense policy and nothing marks which is current, the system will retrieve confidently from the wrong one. It is doing its job. No amount of model quality compensates for a corpus with no notion of currency.
Chunk on structure rather than character count, which is the cheapest large improvement available. Add reranking: retrieve a generous candidate set, then have a model score each candidate against the question and keep the best few, which is consistently one of the largest quality gains per unit of effort. Use hybrid retrieval combining semantic and keyword, because exact identifiers matter more in enterprises than in consumer search.
Exploit metadata such as recency, document type, author, department and status, because a three year old superseded draft and the current signed policy are not equally good answers and only your metadata knows the difference. And handle the question before retrieving it, since expanding an ambiguous question or splitting a compound one into parts often retrieves better than any index change.
Almost every disappointing deployment we see has this in common: nobody could say how good it was. It was judged by whoever tried it last, on whatever they happened to ask.
A workable evaluation practice needs three things. The first is a question set that reflects real usage: fifty to two hundred real questions from real employees, with the correct source document identified for each. Building this is a day or two of work with the teams who will use the system, and it is the highest value day in the project. Include the questions the system should refuse, meaning things not in the corpus and things the asker is not entitled to.
The second is measuring retrieval separately from generation. These are different failures with different fixes, and evaluating end to end tells you something is wrong without telling you what. For retrieval, measure whether the correct passage appeared in the retrieved set at all and how highly it ranked. For generation, measure whether the answer is supported by the passages it cites.
The third is regression testing on every change. Model swaps, chunking changes, connector updates and prompt edits all move quality, sometimes downward. Run the question set on every change, because without this you are tuning blind and your improvements are anecdotes.
Groundedness deserves its own note. An answer can be entirely correct and still not supported by the retrieved passages, which means the model knew it from training rather than from your corpus. That is a failure even when the answer is right, because it will be wrong on the next question and you will have no signal.
Consistently enough to list.
The demo used the good documents. A curated corpus of clean, current, well structured material produces an impressive pilot that tells you nothing about the real store, which contains drafts, duplicates and a decade of superseded versions.
Permissions were left until after the pilot, and then it turns out the connector cannot expose per document access lists and the architecture needs rework. Retrieval quality was never measured, so the team tuned prompts and swapped models while the actual problem was chunking. Connector effort was underestimated, because it is systems integration rather than AI work and is often estimated by the wrong people.
Nobody owned the corpus. Search makes the state of your knowledge base visible in a way nothing else does, and if nobody is responsible for currency and duplication, quality decays and users stop trusting the system. Trust does not come back easily.
And adoption was assumed. Employees who have been failed by three previous search tools will try the new one twice. If both attempts miss, you will not get a third.
The distinctive question in enterprise search is not cost, it is that the corpus is the most sensitive body of text the organisation has. Every contract, strategy document, personnel record and unannounced result, assembled into one queryable index. That is a concentration of risk that did not previously exist in one place.
This makes deployment architecture a first order decision rather than an infrastructure preference. For many organisations a managed cloud service is entirely appropriate, with the right contractual terms about training and retention. For others, in regulated sectors, in defence, in anything covered by a strict data residency obligation, the corpus cannot leave the building, and that constraint is absolute rather than negotiable.
Self hosting is genuinely viable now in a way it was not two years ago. Open weight models have improved to the point where a well built retrieval system with a mid sized local model outperforms a weak retrieval system with a frontier model, because retrieval quality dominates. This is what we build at Aptibit when residency requires it: retrieval pipelines with semantic and hybrid search and citation tracking, running on local vector databases and self hosted inference, so that documents and their embeddings stay on infrastructure the customer controls.
The honest trade off is that self hosting costs you the frontier model's reasoning quality on hard synthesis, and buys you the ability to point at a machine and say the data is there. Which matters more is a question about your obligations rather than about your architecture.
Three components, and the proportions surprise people.
Connector and integration work is normally the largest, and it is conventional systems integration rather than AI engineering. Corpus preparation is the one nobody budgets: deduplication, marking currency, fixing metadata, and deciding what should not be indexed at all. This is not glamorous and it determines the outcome.
Inference and infrastructure is the smallest and the most discussed. Embedding a corpus once is cheap, but re embedding it every time you change chunking is not, so allow for several passes. Query time inference scales with usage and is generally modest against the labour it displaces.
The cost that is never quantified and matters most is the cost of a system people do not trust. A search tool that fails on the first two questions is abandoned, and the second attempt at the same project faces an organisation that has already decided this does not work.
RAG enterprise search earns its place on large, fragmented corpora across many systems where nobody knows where anything is; on high volume repeated questions with a definite answer, such as policy, process and product detail; in support and onboarding where the same questions recur and answers exist but are buried; and in any organisation where senior people spend real time answering questions a document already answers.
It disappoints on small or well organised corpora where existing search already works; on corpora with no notion of currency, where the system confidently retrieves superseded material; in environments where per document permissions cannot be exposed by the source systems; in deployments with no evaluation set, which cannot be improved except by guesswork; and on questions requiring synthesis across many documents at once, which remains genuinely hard.
Enterprise search is the ability to find information held inside your own organisation, across the many systems that hold it, from a single place. It differs from web search in that the content is private, permissions must be respected per user, and the corpus is far smaller but far messier, with duplicates, drafts and superseded versions that no public index has to contend with.
They are the software systems that index internal content and serve queries against it. Classical ones such as Elasticsearch and OpenSearch are built on keyword indexes and return ranked documents. Newer ones add semantic retrieval and a language model to compose direct answers with citations. Many organisations now build their own on top of a vector database and an open or hosted model, because the connector and permission work is bespoke regardless of which product you buy.
This is the wrong first question, and it is the one most buyers ask first. Model choice matters far less than retrieval quality: a strong retrieval pipeline with a mid sized model reliably beats a weak pipeline with the best available model, because a model cannot answer from a passage that was never retrieved. Decide your deployment constraints first, get retrieval right, and treat the model as a component you can swap later.
Enterprise search is the goal of finding internal information. Retrieval augmented generation is the current technique for doing it well: retrieve relevant passages, then have a language model compose an answer grounded in them. RAG is how modern enterprise search is built rather than a competing category.
You need somewhere to store and search embeddings, which can be a dedicated vector database or vector capability added to a database you already run. For most corpora the choice is not performance critical, and it matters far less than chunking, hybrid retrieval and reranking. Choose based on what your team can operate.
By enforcing permissions at retrieval rather than at generation. Store each chunk with the access control list of its source document, filter retrieval to what the asking user may see, keep those permissions synchronised as they change, and log which passages were returned to whom. A language model has no concept of authorisation and will summarise whatever retrieval gives it, so the boundary has to sit earlier in the pipeline.
Build a set of fifty to two hundred real questions with the correct source document identified for each, including questions the system should refuse. Measure retrieval and generation separately: for retrieval, whether the right passage was returned and how highly it ranked; for generation, whether the answer is actually supported by the passages cited. Re run that set on every change, because model swaps and chunking changes both move quality in directions you cannot predict.
Yes, and it is considerably more practical than it was two years ago. Open weight models plus a local vector database and self hosted inference keep documents, embeddings and queries inside your network, which is often the only way to satisfy a strict data residency obligation. You trade some reasoning quality on hard synthesis for the ability to demonstrate exactly where the data lives.
It depends almost entirely on connector count and corpus condition rather than on the AI work. A single well organised source with clean permissions is a short project. Six systems with different permission models and a corpus nobody has curated is a long one. Estimate from the integration and corpus work and treat the retrieval pipeline as the predictable part.