Agentic RAG moves beyond single-pass document lookup. The system plans what to retrieve, reformulates queries when initial results are insufficient and synthesises multi-source answers. This article explains what that means and when to use it.
Standard RAG is a single-pass process: receive a query, retrieve relevant chunks, generate an answer. Agentic RAG introduces a reasoning loop between the query and the answer — the system can evaluate whether its initial retrieval was adequate, decide to search again with a refined query, break a complex question into sub-questions and synthesise results from multiple retrieval passes. This is a meaningful architectural step up from standard RAG, with distinct use cases, benefits and risks.
In a standard RAG pipeline, retrieval is stateless and single-pass. The query is embedded, the top-K chunks are returned and the language model generates from that context. If the initial retrieval missed a relevant document or the question required synthesising information from disparate sources, the system has no mechanism to compensate.
Agentic RAG adds a planning and evaluation layer — often implemented by a language model acting as an orchestrator — that can:
The technical grounding is described by Anthropic as a system that adds a "reasoning/planning loop on top of retrieval" — the agent decides what to retrieve, not just how to search once.
Many enterprise queries do not reduce to a single document lookup. A compliance officer asking "what are our current obligations under the Privacy Act for this customer scenario?" may need information from the Privacy Act summary, the organisation's data handling policy, the relevant consent records and recent OAIC guidance — each from a different source. Standard RAG makes one retrieval pass and hopes the right content surfaces. Agentic RAG plans a retrieval strategy, executes it across multiple sources and composes a complete answer.
BCG's 2025 research found that AI agents accounted for around 17% of total AI value in 2025 and are projected to reach 29% by 2028, reflecting increasing enterprise confidence in systems that can handle multi-step reasoning tasks.
A typical agentic RAG architecture includes:
Orchestrator (reasoning layer): A language model — often a capable reasoning model — that receives the original query, plans the retrieval strategy, executes sub-queries and evaluates whether the accumulated context is sufficient to generate a final answer.
Retrieval tools: The orchestrator calls retrieval tools as needed. These may include vector search over internal knowledge bases, keyword search, structured database queries or external API calls. Each retrieval tool is defined with a clear interface the orchestrator can invoke.
Query decomposition: For complex questions, the orchestrator breaks the query into sub-questions, retrieves separately for each and combines the results before final generation.
Self-evaluation: After each retrieval step, the orchestrator assesses whether the retrieved context is sufficient — a "critic" step. If not, it reformulates the query or retrieves from a different source.
Context window management: Across multiple retrieval steps, the accumulated context must be managed to fit within the language model's context window, often requiring summarisation of earlier retrieved content.
Agentic RAG is more complex to build, test and monitor than standard RAG. Each additional reasoning step introduces latency and cost — a system that makes three retrieval calls and two intermediate reasoning steps will take measurably longer and cost more per query than a standard single-pass retrieval.
The failure modes are also more complex. An error in the first sub-query can propagate through subsequent steps, producing a confident but compoundly wrong final answer. This makes evaluation harder: you need to test not just final answer quality but intermediate retrieval and reasoning steps.
For organisations considering agentic RAG, the practical guidance is to start with standard RAG for the majority of use cases and introduce agentic retrieval only for the specific query types that demonstrably require it — typically complex analytical questions, multi-source synthesis and tasks that require disambiguation before retrieval. Edison AI's AI implementation team typically designs hybrid pipelines where simple queries route to standard RAG and complex queries route to an agentic retrieval path, with query classification determining the routing decision.
Evaluate whether your current highest-value RAG use cases involve complex, multi-source questions or simple factual lookups. For the latter, standard RAG with hybrid search and re-ranking is usually sufficient. For the former, prototype an agentic retrieval path on a bounded, well-governed knowledge base. Measure latency, cost per query and answer quality against your standard RAG baseline before committing to an agentic architecture at scale.
Edison AI builds bespoke AI systems — including retrieval over your own documents — for Australian businesses.
Standard RAG runs one retrieval pass — embed the query, fetch the top-K chunks, generate an answer. Agentic RAG adds a reasoning loop: the system can assess whether the initial retrieval was sufficient, reformulate the query, retrieve again from different sources, break complex questions into sub-queries and synthesise a final answer from multiple retrieval steps.
Questions that require synthesising information from multiple documents, multi-step reasoning, or disambiguation before retrieval are candidates for agentic RAG. Examples: comparing two policy versions, answering a question whose answer depends on first resolving a related sub-question, or searching across multiple knowledge domains to compile a complete response.
Agentic RAG introduces variable latency (more retrieval steps take more time), higher cost (more LLM calls per query) and compounding error risk — a poor initial sub-query can propagate through subsequent reasoning steps. It also requires careful guardrails to prevent the agent from making external calls or taking actions outside its intended scope.
Edison AI helps Australian businesses move from AI curiosity to practical implementation, with workflow design, team training and measurable outcomes. Tell us about your setup and we'll come back with a sequenced plan grounded in the same thinking you just read.
Article: Agentic RAG: When Retrieval Systems Start to Reason
