Why RAG is not a substitute for information architecture

Retrieval-augmented generation has become the default answer to “how do we make the model know our business.” Chunk the documents, embed them, retrieve at query time, generate. The architecture diagram fits on a napkin, and that is precisely the problem: it makes retrieval look like a solved step instead of the load-bearing one.

I spent years building search for one of the most information-hostile industries there is — industrial and chemical product data — and the lesson transfers directly: retrieval quality is decided by the structure of the information, long before any model sees it.

Consider what the napkin diagram skips:

Entity resolution. If the same product, customer, or policy appears under five names across your document base, similarity search returns five partial views and the model synthesizes a confident blur. No embedding model resolves identity for you; that is data work.

Authority and freshness. Enterprises do not have “documents”; they have drafts, superseded versions, regional variants, and one canonical answer living in somebody’s inbox. Retrieval that cannot rank authority returns the obsolete policy with perfect semantic similarity.

Structure that already exists. Half of enterprise knowledge is tables, hierarchies, and fields — information that is computable, not just retrievable. Flattening a rate table into prose chunks so a model can guess at what a lookup would have answered exactly is architectural malpractice.

The questions people actually ask. Real query loads mix lookup (“what is the limit for X”), aggregation (“how many of our contracts have Y”), and judgment (“should we do Z”). Only the first is a natural retrieval problem. Systems that route everything through similarity search answer the other two categories fluently and wrongly.

None of this means RAG is wrong. It means RAG is the last mile. The work that determines whether it succeeds is classical, unfashionable information architecture: canonical entities, authority ranking, structured extraction, taxonomies, and a routing layer that knows which questions deserve a database query instead of a vector search.

Vertical AI wins inside the workflow, not at the chatbot layer — and the workflow runs on structured information. Teams that skip the structure to get to the demo faster are not saving time. They are borrowing it.