Loading…
Loading…
Enterprise search with RAG, explained for leaders: why keyword search falls short, how AI answers work, how to protect permissions and why projects stall.
The model is the commodity. Finding the right passage is the product. Four of the five stages in an AI search system are about retrieval, yet most budgets are spent the other way round.
Keyword search failed because people and documents use different words. It still wins on exact codes such as invoice numbers and error messages, so a serious system keeps both.
Permissions are the hardest part, and there is no elegant fix. A language model has no sense of who may see what.
Build a test set of fifty to two hundred real questions with known answers. Measure search and answering separately, because they fail for different reasons.
Fix your content first. A system that answers confidently from an outdated policy is working correctly and still giving the wrong answer.
Search for enterprise search and nearly every result is a vendor: Elastic, Glean, Coveo, Sinequa, Moveworks, IBM, Hyland, Stravito. Most make good products and publish accurate material.
The most telling result is a community discussion in which an employee of a mid sized company describes finding information as a daily scavenger hunt across Slack, Google Drive, Notion, Jira and old email threads. It ranks because it is written by someone with the problem, not someone selling the answer.
This guide is written from the builder's side. At Aptibit, we design and build these systems, and we know where an impressive demo meets the reality of a real document store.
Enterprise search is the ability to find information held inside your organisation, across every system that holds it, from one place. That definition has not changed in twenty years. What finding means has.
Classic enterprise search returns documents. You type words, it lists files containing those words, and you read until you find the answer. This is keyword search.
AI search built on retrieval augmented generation (RAG) returns answers. You ask in your own words. The system finds the passages most likely to hold the answer, and a language model writes a reply from them, with links back to the sources.
Commercially, the difference is who does the reading: the first leaves it to the employee, the second does it for them. The catch is that the second can be confidently wrong in a way the first cannot. That is why the engineering around it must be more careful, not less.
The reasons matter, because they define what AI search has to fix.
The first is mismatched vocabulary. Someone searches for time off policy, the document says annual leave entitlement, and nothing matches. Every organisation has its own dialect, and keyword search expects you to know it already.
The second is that keywords have no idea what a question is. How many days' notice do I need before leave? To a keyword index, that is a bag of words, most of them noise.
The third is ranking without context. A keyword engine does not know that the current policy outranks a three year old email that happens to repeat the same words. The fourth is fragmentation: the rule sits in one system, the exception in a chat thread and the latest version in someone's personal folder.
Keyword search still wins on exact identifiers: invoice numbers, part codes, error messages, ticket references. When someone types an exact code they want that code, not things that are conceptually similar. Serious systems use both methods together, which is what hybrid search means.
There are five stages. The failures that matter cluster in the first two.
Connecting. Connectors pull content from document stores, wikis, ticketing tools, chat, email and databases. This is plain integration work, and it is usually the largest single cost, because every system has its own login, limits and permission rules.
Splitting. Documents are cut into passages small enough to find precisely and large enough to make sense alone. Too small, and the passage loses its meaning. Too large, and the useful part is buried. Splitting along the document's own sections and headings beats cutting at a fixed length, and often improves answers more than a model upgrade.
Indexing. Each passage is converted into a numerical fingerprint of its meaning. Passages that mean similar things get similar fingerprints, so time off policy can find annual leave entitlement without anyone writing a list of synonyms.
Searching. The question is converted the same way and matched against the index. A strong system combines search by meaning with search by keyword, then reranks the results: a second model scores each candidate against the question and keeps the best. Reranking is one of the highest value additions available, and it is often skipped.
Answering. The best passages and the question go to a language model, which writes an answer and cites the passage behind each claim. Four of the five stages are about finding the right material. Only one is the model.
This is the hardest part of enterprise search. It has no elegant solution, and it is why many pilots never reach production.
Not everyone may see everything: salaries, disciplinary records, unannounced results, legal advice. Your source systems already enforce this. The search layer must reproduce it exactly, even though it has just read everything.
When this fails, it fails quietly. A model will summarise any passage it receives for whoever asked. If search returns something the person should not see, the system will paraphrase a confidential document for them, fluently and with a citation.
There are two broad approaches. The first tags every passage with its source document's access rules and limits each search to what the person may see. It stays correct as permissions change, so it is the right default, though it adds a check to every query. The second keeps separate indexes for each security boundary. It is simpler and faster, but only works when boundaries are few and stable.
Three lessons are usually learned the hard way. Permissions change, so the index must keep up; a weekly refresh means a week in which a former project member can still find that material. Each passage should inherit the document's access rules, not the folder's, because documents are moved and shared individually. And log every search result, not just every question, so you can answer later what was shown to whom.
If a vendor cannot explain their permission model in detail, that is the most important thing you will learn in the meeting.
The most common cause of a disappointing system is not the model. The right passage was never found. The model then either says it does not know, which users read as broken, or improvises from nearby material, which is worse.
Fix the content first. If your store holds four versions of the expense policy and nothing marks the current one, the system will answer confidently from the wrong one. No model can make up for content with no sense of which version is current.
Then make the proven improvements. Split documents along their structure. Add reranking. Combine search by meaning with search by keyword, because exact codes matter more inside a business than on the open web.
Use the information you already hold about each document: date, type, author, department and status. Only that information tells the system that a signed policy beats a three year old draft. And clarify the question before searching, for example by splitting a two part question into two.
Disappointing deployments tend to share one trait: nobody can say how good the system is. It is judged by whoever tried it last, on whatever they happened to ask.
A sound evaluation practice has three parts. First, a question set drawn from real use: fifty to two hundred real questions from real employees, each with the correct source identified. Building it takes a day or two with the teams who will use the system, and it is the most valuable time in the project. Include questions the system should decline, because the answer is not there or the person is not entitled to it.
Second, measure search and answering separately. They fail for different reasons and need different fixes. For search, check whether the right passage was found and how high it ranked. For answers, check whether each claim is supported by the cited sources.
Third, rerun the set after every change. Model swaps, splitting changes, new connectors and prompt edits all move quality, sometimes down. Without this, improvements are anecdotes.
One point deserves emphasis. An answer can be correct yet not supported by the passages found, because the model knew it from its general training. Treat that as a failure. The next answer may be wrong, and you will have no warning.
The pattern is consistent enough to list.
The demo used the best documents. A hand picked set of clean, current material makes an impressive pilot and says nothing about the real store, with its drafts, duplicates and years of outdated versions.
Permissions were left until after the pilot, and the connector turned out unable to expose document level access rules. Search quality was never measured, so the team tuned prompts and swapped models while the real problem was how documents were split. Connector effort was underestimated, because it is integration work estimated as if it were AI work.
Nobody owned the content. Search exposes the state of your knowledge base like nothing else. If nobody is responsible for keeping it current and removing duplicates, quality decays and trust goes with it.
And adoption was assumed. People let down by three earlier search tools will try a new one twice. If both attempts miss, there is no third.
The defining question here is not cost. It is that the search index becomes the most sensitive collection of text the organisation owns: every contract, strategy paper, personnel file and unannounced result in one searchable place.
That makes where it runs a board level decision, not an infrastructure preference. For many organisations a managed cloud service is right, with clear contract terms on training and data retention. For regulated sectors, defence and anyone bound by strict data residency rules, the content cannot leave the building.
Running it yourself is now genuinely practical. Openly available models have improved so much that a well built search layer with a mid sized local model outperforms a weak one with the most advanced model. At Aptibit, this is what we build when residency demands it: AI search combining meaning and keyword search, with source citations, running entirely on infrastructure the client controls.
The trade is clear. Self hosting gives up some reasoning power on the hardest questions. In return, you can point to a machine and say: the data is there. Which matters more depends on your obligations.
There are three components, and the proportions surprise people.
Connecting systems is usually the largest, and it is integration work rather than AI work. Preparing content is the one nobody budgets for: removing duplicates, marking what is current, fixing document details and deciding what should never be indexed. It is unglamorous, and it decides the outcome.
Computing power is the smallest cost and the most discussed. Indexing your content once is cheap, but you will reindex each time you change how documents are split, so allow for several passes. Running costs grow with use and are usually modest against the time saved.
The cost that matters most is rarely counted: a system people do not trust. A tool that misses the first two questions is abandoned, and the next attempt faces people who have already decided it does not work.
AI enterprise search earns its place when information is spread across many systems and nobody knows where anything lives; when the same questions with definite answers come up again and again, such as policy, process and product detail; in support and onboarding, where answers exist but are buried; and wherever senior people spend real time answering questions a document already answers.
It disappoints when content is small or well organised and existing search already works; when nothing marks which version is current; when source systems cannot expose document level permissions; when there is no test set, so improvement is guesswork; and on questions that need synthesis across many documents at once, which remains genuinely hard.
Enterprise search is the ability to find information held inside your own organisation, across all the systems that hold it, from one place. It differs from web search in three ways. The content is private. Each person may only see what they are entitled to. And the collection is smaller but far messier, full of duplicates, drafts and outdated versions.
They are the software systems that index internal content and answer queries against it. Classic engines such as Elasticsearch and OpenSearch match keywords and return a ranked list of documents. Newer systems add search by meaning and a language model that writes a direct answer with sources. Many organisations now build their own, because connecting their systems and respecting their permissions is bespoke work whichever product they buy.
It is the question most buyers ask first, and it is the wrong first question. The model matters far less than finding the right passage. A strong search layer with a mid sized model reliably beats a weak one with the best model available, because no model can answer from a passage it never received. Settle where the system must run, get retrieval right, and treat the model as a part you can swap later.
Enterprise search is the goal: finding internal information. Retrieval augmented generation (RAG) is today's best technique for reaching it. The system first retrieves the relevant passages, then a language model writes an answer based only on them. RAG is how modern enterprise search is built, not a competing category.
You need somewhere to store and search the numerical fingerprints of meaning that AI search relies on. That can be a dedicated vector database or a vector feature added to a database you already run. For most organisations the choice is not critical. It matters far less than how documents are split, how search is combined and how results are reranked. Choose what your team can run well.
Enforce permissions at the search step, not at the answer step. Tag every passage with the access rules of its source document, limit each search to what the person asking may see, keep those rules in step as they change, and log which passages went to whom. A language model has no sense of who is allowed to see what. It will summarise whatever it is given, so the boundary must sit earlier.
Build a set of fifty to two hundred real questions, each with the correct source document identified, including questions the system should decline. Measure search and answering separately. For search, check whether the right passage came back and how high it ranked. For answers, check whether each claim is supported by the sources cited. Rerun the set after every change, because model swaps and setting changes move quality in ways nobody can predict.
Yes, and it is far more practical than it was two years ago. Openly available models, a local search index and inference on your own servers keep documents and queries inside your network. That is often the only way to meet a strict data residency rule. You give up some reasoning quality on the hardest questions, and you gain the ability to show exactly where your data lives.
It depends mostly on how many systems you connect and how tidy your content is, not on the AI work. One well organised source with clean permissions is a short project. Six systems with different permission models and uncurated content is a long one. Plan from the integration and content work, and treat the AI layer as the predictable part.