Retrieval-augmented generation is the default architecture for putting a language model to work on an organisation's own documents. The idea is straightforward: when a user asks a question, the system retrieves relevant passages from a document store and asks the model to answer from them. Policies, contracts, maintenance manuals, circulars and case files become searchable in plain language.
Enterprise RAG is also where many generative AI programmes stall. The pilot, built on a few hundred clean documents and questions written by the project team, performs well. The production system, facing hundreds of thousands of documents in mixed formats, users who phrase questions their own way, and access rules that differ by department, does not. Answers miss documents that exist, cite superseded versions, or, worse, reveal content the user was never entitled to see.
This piece sets out what research has found about why RAG systems fail, proposes a failure taxonomy across four layers (ingestion, retrieval, generation and permissions), and describes how to diagnose and fix a system using it. The aim is a shared vocabulary for engineering teams, risk owners and business sponsors, so that "the chatbot is wrong" becomes a fault that can be located.
Why the enterprise RAG demo always works
A RAG pilot is usually tested under conditions that hide the hardest problems. The documents are selected, so they are clean, current and relevant. The questions are written by people who know what the documents contain, so they use the same vocabulary. Everyone on the pilot can see everything, so permissions never arise. And evaluation is often informal: the team reads answers and judges them plausible.
Production removes all four protections at once. Documents arrive as scanned PDFs, spreadsheets, slide decks and email threads, often in several versions. Users ask about things using internal shorthand, regional language or simply different words. Access control becomes the dominant design constraint. And a plausible answer is not the same as a correct one.
The result is a familiar pattern in enterprise AI. McKinsey's State of AI 2025 survey found that 88% of organisations use AI regularly in at least one function, but only about one-third have begun to scale AI programmes. Menlo Ventures' December 2025 report found that 47% of enterprise AI deals reach production, up from 25% a year earlier. Progress is real, but the gap between pilot and production remains the main obstacle, and document question-answering systems are among the most common casualties.
What the research says about RAG failure
The most useful empirical starting point is "Seven Failure Points When Engineering a Retrieval Augmented Generation System" by Scott Barnett and colleagues (arXiv, January 2024). The authors drew on three case studies in research, education and biomedical domains, including a biomedical question-answering experiment over a corpus of several thousand scientific documents, and identified seven failure points:
Missing content: the question cannot be answered from the documents, but the system answers anyway. Missed the top-ranked documents: the answer exists but ranks too low to be returned. Not in context: relevant documents are retrieved but dropped when results are consolidated into the prompt. Not extracted: the answer is in the context, but the model fails to extract it, often because of noise or contradictions. Wrong format: the model ignores a requested format. Incorrect specificity: the answer is too general or too specific. Incomplete: the answer omits information available in the context.
The paper's two lessons are as important as the list: that "validation of a RAG system is only feasible during operation", and that "the robustness of a RAG system evolves rather than designed in at the start." Both point to the same conclusion. A RAG system needs production monitoring and a way to classify failures, not just a pre-launch test.
A second finding concerns how models use what they retrieve. In "Lost in the Middle: How Language Models Use Long Contexts" (Nelson Liu and colleagues, first posted in July 2023 and accepted by Transactions of the Association for Computational Linguistics), the authors found that "performance is often highest when relevant information occurs at the beginning or end of the input context", and degrades when the relevant passage sits in the middle of a long context. Simply retrieving more passages can make answers worse.
Barnett's list is a strong foundation, but it was derived from systems where access control was not the central concern. In most enterprises, it is. The taxonomy below extends the research to cover ingestion and permissions explicitly.
A failure taxonomy for enterprise RAG
The taxonomy groups failures by the layer where they originate, because that is where they must be fixed. A failure that appears in the answer often begins much earlier. Each entry is mapped, where relevant, to Barnett's failure points (FP) or to the OWASP Top 10 for LLM Applications 2025.
Three patterns emerge from the taxonomy. First, ten of the twenty failure modes sit in ingestion and retrieval, before the model generates anything, and three of the five permission failures are fixed in the index rather than in the model. Teams that respond to poor answers by changing the model or the prompt are often working on the wrong layer. Second, retrieval failures are frequently ingestion failures in disguise: wrong-scope answers (R4) cannot be fixed without the metadata captured at I4. Third, the permissions layer is not an add-on. It shapes how the index is built, how queries are filtered, and what can be cached.
Permissions deserve their own layer
The OWASP Top 10 for LLM Applications 2025 lists vector and embedding weaknesses (LLM08) as a distinct risk, and its description reads like an enterprise RAG incident report. It warns that "inadequate or misaligned access controls can lead to unauthorized access to embeddings containing sensitive information", and that in multi-tenant environments where several classes of users share the same vector database, context can leak between users or queries. Its first recommended mitigation is to "implement fine-grained access controls and permission-aware vector and embedding stores", with datasets partitioned by access level.
The design choice that matters most is where permission checks happen. If documents are indexed by a service account with access to everything and the system relies on the model to withhold restricted content, the system is only as secure as the model's restraint. The more robust pattern is to carry each document's access labels into the index and filter retrieved passages by the requesting user's identity before anything reaches the model. The model never sees what the user cannot see.
Permission drift (P2) is the quiet failure. Access is revoked in the document management system when someone changes role, but the RAG index, synchronised weekly, still answers their questions from documents they can no longer open. For organisations subject to India's Digital Personal Data Protection Act, a RAG system that surfaces personal data to people without a purpose to see it is a data protection problem, not only a security one.
The final risk, aggregation exposure (P5), has no complete technical fix. A model that summarises across a user's permitted documents can still produce a combination that is sensitive, such as an inferred restructuring plan. In high-risk domains, classifying sensitive topics and routing those queries to human review is the practical control.
Diagnosing a failing system
The taxonomy is most useful as a diagnostic procedure. A practical sequence:
1. Collect real failures. Gather a sample of queries where users reported poor answers or where answers were rejected in review. A hundred is enough to begin. 2. Trace each one. For each query, log what was retrieved, what reached the prompt and what the model produced. Without this trace, diagnosis is guesswork. 3. Classify by first failing layer. Was the answer in the corpus at all (I1, R5)? Was it parsed correctly (I1, I2)? Was it retrieved (R1, R2)? Did it reach the prompt (R3)? Did the model use it (G1, G2)? Did the user have the right to see what was used (P1 to P5)? 4. Fix the most common first failure. Fixing a downstream symptom while an upstream cause persists produces fragile improvements. 5. Repeat in production. As Barnett and colleagues observed, robustness evolves rather than being designed in. The classification should become a standing part of operations.
Evaluation that matches the taxonomy
Evaluation should measure each layer separately. The Ragas framework (Shahul Es and colleagues, arXiv, first submitted September 2023) proposes assessing RAG systems on three dimensions: the ability of retrieval to identify relevant and focused passages, the ability of the model to exploit those passages faithfully, and the quality of the generation itself. The taxonomy adds two more: ingestion fidelity and permission correctness.
Permission tests deserve particular emphasis because they are rarely in standard evaluation suites. A useful routine is to create test identities with known, limited access and run a standing set of queries designed to surface restricted content. Any hit is a severity-one defect. A separate PanScience Research piece covers evaluation before deployment more broadly.
What changes for Indian enterprises
Several failure modes in the taxonomy are sharper in Indian deployments. Parsing loss (I1) is more common where large parts of the corpus are scanned documents: stamped approvals, signed agreements, older files digitised from paper, and records in regional scripts that general-purpose OCR handles unevenly. A team that measures extraction fidelity only on born-digital English PDFs will overestimate how much of its corpus the system can actually read.
Vocabulary mismatch (R1) is also amplified. Employees and customers mix English with Hindi or another regional language in the same question, use transliterated terms, and rely on internal acronyms that never appear in formal documents. Hybrid keyword and semantic search, a maintained glossary of internal terms and query rewriting tested on real, mixed-language questions matter more here than a change of embedding model.
Version control (I3) is a third pressure point. Regulated businesses work from circulars, master directions and internal policies that are amended frequently, often with an effective date that differs from the publication date. An index that does not record which version applied when will give answers that were correct last quarter. For compliance use cases, the effective-date metadata captured at ingestion is as important as anything the model does.
Deployment choices and where on-premise fits
For regulated organisations, the permission and data-residency requirements of enterprise RAG often push towards private or on-premise deployment, where documents, indexes and logs remain inside the organisation's own environment. That choice simplifies some problems, notably derived data leakage to external services, but does not solve the permission-aware indexing or drift problems, which are architectural. A separate PanScience Research piece covers the economics of on-premise AI.
PanScience's portfolio includes Insituate AI, which describes itself as an on-premise enterprise RAG and agent platform (company-reported). The relevant lesson from platforms of this kind is that the hard engineering in enterprise RAG lies less in the model than in connectors, parsing, permission synchronisation and logging, the parts of the taxonomy that sit before and around generation.
Conclusion
Enterprise RAG fails in production for reasons that are mostly knowable in advance. Documents are parsed badly, chunked carelessly or indexed in several versions. Retrieval misses what users mean or what the metadata could have filtered. Models ignore, misread or embellish what they are given. And permissions, the layer pilots never test, turn a useful system into a data leak.
The fix is not a better model. It is a discipline: trace every answer, classify failures by the layer where they start, fix upstream causes first, and test permissions as rigorously as accuracy. Organisations that adopt a shared failure vocabulary will find that their RAG systems improve steadily in production, which is, as the research suggests, the only place they can.
Methodology and sources
The seven failure points and the two lessons are from Barnett, Kurniawan, Thudumu, Brannelly and Abdelrazek, "Seven Failure Points When Engineering a Retrieval Augmented Generation System" (arxiv.org/abs/2401.05856, January 2024). The position effect is from Liu and colleagues, "Lost in the Middle: How Language Models Use Long Contexts" (arxiv.org/abs/2307.03172). The evaluation dimensions are from Es and colleagues, "Ragas: Automated Evaluation of Retrieval Augmented Generation" (arxiv.org/abs/2309.15217). Vector and embedding risks and mitigations are from OWASP's LLM08:2025 entry (genai.owasp.org/llmrisk/llm082025-vector-and-embedding- weaknesses) and the OWASP Top 10 for LLM Applications 2025 list (genai.owasp.org/llm-top-10). Adoption figures are from McKinsey, The state of AI in 2025 (mckinsey.com/capabilities/tech-and- ai/our-insights/the-state-of-ai-2025) and Menlo Ventures' State of Generative AI in the Enterprise 2025 (menlovc.com), as verified by the editor. The four-layer taxonomy, its twenty failure modes, the mappings and the diagnostic procedure are PanScience Research's analytical contribution, synthesised from these sources and from common engineering practice; they are not derived from a measured dataset. The Insituate AI description is company-reported from panscience.xyz.