Measure Before You Optimize: A Latency Budget for Production RAG
- Ignas Vaitukaitis
- 57 minutes ago
- 14 min read
TL;DR:
Vector retrieval eats roughly 35% of time-to-first-token in production RAG, and its P95 can sit 64x above the median while generation stays tight. A measured, per-query latency budget, not a blended average, took one production system's P95 SLO compliance from 30% to 95%. The method: instrument every stage, baseline p50/p95/p99 under real traffic, allocate the end-to-end target by measured share, fix only the stages that overspend, then gate deploys on regressions.
A latency budget for production RAG splits an end-to-end p95 target across embedding, retrieval, reranking, prefill, and decode, with each stage's share set by measurement. You measure before you optimize because the stage teams blame is rarely the one overspending. This guide walks the full method: instrument every stage, baseline the percentiles under real traffic, allocate by measured shares, fix only the overspenders, then gate deploys on regressions. As of September, 2026, the case for per-stage percentiles over blended averages is stark.
The numbers that anchor the method:
Retrieval took roughly 35% of time-to-first-token in a 2024 systems study of RAG inference trade-offs, against the folk wisdom that generation dominates.
A baseline IVF vector index showed a p95 more than 64x above its median in the 2024 EdgeRAG measurements. The tail is where budgets die.
SAGE's per-query retrieval budget, published in August 2026, lifted P95 SLO compliance from 30% to 95% and cut P95 latency from 5.6 to 3.6 seconds.
Halving output tokens cuts roughly 50% of generation latency, while halving input tokens buys only 1 to 5%, per OpenAI's 2026 latency guidance.
What a Latency Budget for Production RAG Is and Why You Measure Before You Optimize
A latency budget for production RAG is a percentile-based allocation of your end-to-end response-time target across every stage of the pipeline: embedding, retrieval, reranking, prefill, and decode each get a measured share of the total, and each share is tracked at p95 or p99 rather than the mean. You measure first because RAG systems rarely miss their targets due to one slow component. They miss because teams optimized components nobody had measured.
That's the whole argument, and it has an uncomfortable amount of evidence behind it.
Averages actively lie in this domain. Google's 2013 analysis of tail latency at scale established that in fan-out systems, the overall latency of a request is bound by the slowest of its parallel calls, which means a healthy-looking mean can hide a tail that ruins one request in twenty. RAG inherits this problem directly: a query fans out to an embedding model, a vector index, often a reranker, then a language model. Any one of them can blow the budget while the dashboard average stays green.
The field has started taking this seriously. The SAGE adaptive-retrieval paper, published in August 2026, targets P95 latency instead of mean latency as its service objective, precisely because SLO compliance and mean latency diverge sharply once real query-distribution variance shows up. A static configuration that looks fine on average met its 5-second P95 SLO only 30% of the time in SAGE's evaluation. A measured, per-query budget hit 95%.
The discipline this article walks through has five moves:
Instrument every stage with per-span timing.
Baseline p50, p95, and p99 under real traffic.
Allocate the end-to-end target across stages by measured proportions.
Optimize only the stages that overspend their share.
Enforce the budget with alerts and regression gates.
Skip the first two and the last three become guesswork. Expensive guesswork, usually.
How Slow Is Too Slow? What Users Actually Tolerate From a RAG Answer
Users tolerate roughly one second before a delay breaks their flow of thought, and around ten seconds before their attention leaves entirely, so a production RAG budget should anchor to those human thresholds rather than a number picked for engineering convenience. The classic figures come from Miller's 1968 work and Card, Moran and Newell's 1983 follow-up: about 0.1 seconds feels instantaneous, about 1 second preserves continuity, and about 10 seconds is where people give up and check something else.
Three findings complicate the folk version of those numbers:
Sub-100ms delay is perceptible. A 2017 study on latency perception in mouse-based interaction found users detect delays well below the classic 100ms threshold, so "under 100ms is invisible" is a comfortable myth.
Task type moderates tolerance. A CHI 2026 paper on response latency in human-LLM interaction found how negatively users react to delay depends on what they're asking for. A quick factual lookup and a research-style synthesis carry different patience budgets.
Visible waiting beats silent waiting. The same 2026 work found that some intentional, surfaced latency doesn't uniformly hurt user perception. A "searching sources…" indicator can buy engineering time without costing trust, as long as the delay is deliberately shown instead of silently endured.
That last point is a genuinely useful loophole. If your retrieval stage needs 800ms you can't yet recover, showing the user what's happening converts dead air into perceived diligence.
The practical anchors: aim near 1 second to first visible token for conversational use, accept a few seconds for search-style tasks, and treat 10 seconds of silence as a failed request regardless of how good the answer is.
Where the Milliseconds Go in a Production RAG Pipeline
A production RAG request spends its time across six separable stages: embedding the query, searching the vector index (often with metadata filtering), reranking candidates, assembling the prompt, LLM prefill over the retrieved context, and LLM decode. Each stage has a distinct latency profile, and the proportions defy the folk wisdom that generation dominates everything.
Retrieval costs more than most teams assume. A 2024 systems study of RAG inference trade-offs measured retrieval alone at roughly 35% of total time-to-first-token, with RAG roughly doubling TTFT compared to a plain LLM call (495ms to 965ms in their setup). Throughput fell by up to 20x as the vector store scaled from 1 million to 100 million chunks.
The tails are worse than the medians suggest. Much worse.
A 2024 study of edge-device RAG indexing found a baseline IVF index whose 95th-percentile query latency exceeded its median by more than 64x.
Meanwhile the encoding and prefill stages in the 2024 trade-offs study showed p99-to-p50 gaps of only 50 to 60ms on HNSW-SQ and IVF-SQ indexes. That asymmetry is the core lesson of this section: retrieval tails diverge from medians by orders of magnitude while generation stages stay comparatively tight, which is why a per-stage view at p95 finds problems a blended average never will.
Two more line items deserve a place in the mental model:
Context length inflates prefill. OpenAI's GPT-4.1 documentation from April 2025 reported TTFT of roughly 15 seconds at 128,000 tokens of context, rising toward a minute at 1 million tokens, on an optimized inference stack. Every extra retrieved chunk you stuff into the prompt buys some of that.
Retrieval policy compounds retrieval cost. The same 2024 trade-offs study found that re-querying the retriever every 4 generated tokens pushed end-to-end latency to roughly 30 seconds, with retrieval and re-prefill eating 81% of the total. The authors called that regime prohibitively expensive for production, and they're right. How often you retrieve belongs in the budget alongside how fast each retrieval runs.
Anyone who has profiled one of these pipelines knows the pattern: the stage you suspected is rarely the stage that's guilty.
What You Need Before You Build the Budget: Tracing, Real Traffic, and Percentiles
Building a latency budget takes four things up front: tracing that understands GenAI operations, production-like query traffic, percentile histograms for every stage, and a quality baseline to check latency wins against. Skip any of these and the steps that follow produce numbers you can't trust.
Here's the checklist:
OpenTelemetry GenAI semantic conventions. The OpenTelemetry GenAI Special Interest Group, formed in 2024, defines standard span attributes for LLM calls, and the conventions have since grown to cover retrieval spans and data-source attributes. That last part matters for RAG: you want the vector search traced with the same vocabulary as the model call, in the same trace.
Real traffic, or the closest thing you have. Tail behavior is a property of the query distribution. Synthetic test queries that all land in dense index regions will show you a tail that doesn't exist and hide the one that does.
Percentile histograms per stage. Store p50, p95, and p99 for each stage separately. A blended end-to-end mean is the exact instrument this whole method exists to replace.
A quality baseline. A November 2025 paper on rarity-aware RAG evaluation argues that latency should be decomposed into embed, retrieve, and rerank components and reported alongside answer quality instead of as a disconnected system metric. If you can't detect a quality drop, every latency optimization looks free.
One caveat worth knowing before you start. As of OpenTelemetry's 2026 guidance on GenAI observability, the specification defines how to record durations but prescribes no latency thresholds. The SLO is yours to set. That gap is the point of the next five steps.
Step 1: Instrument Every Stage From Query to Rendered Answer
Step 1 adds a timed span to each pipeline stage, from the moment the query arrives to the moment the answer renders, so every later decision rests on per-stage data. The OpenTelemetry GenAI semantic conventions give you a shared vocabulary for this, including a gen_ai.client.operation.duration histogram metric built for exactly the p50-versus-p99 comparisons the budget needs.
Work through the pipeline in order:
In your query handler, open a parent span covering the full request lifetime.
Wrap the embedding call in a child span. It will read as trivially fast. Record it anyway, because "trivially fast" is a claim you're about to test.
Wrap the vector search in a retrieval span, attaching the data-source attributes the GenAI conventions define, plus the k requested and any metadata filters applied. Filtered queries deserve their own dimension.
Wrap the reranker in its own span with the candidate count as an attribute.
For the generation call, record two numbers: time to first token, and total duration. In a streaming setup these diverge wildly, and users experience the first one.
Close the parent span when the final token renders client-side, so network and post-processing time can't hide.
Once deployed, a single request should produce a waterfall trace where each stage's duration is visible at a glance. If two stages overlap in the trace, good. That's concurrency you'll want to protect later.
The streaming distinction is the part teams most often skip. A pipeline reporting only total duration will tell you a 6-second streamed answer and a 6-second silent stall are the same event. They are nowhere near the same event.
Step 2: Measure Baseline p50, p95, and p99 Under Real Traffic
Step 2 collects per-stage percentiles under your real query distribution for long enough to see the tails, before anyone touches a config. Run at least a full traffic cycle. Weekday and weekend query mixes differ, and the tail lives in the mix.
Expect the stages to behave differently at the tail. Vector-index latency distributions are famously skewed: the 2024 EdgeRAG study measured a baseline IVF index whose P95 sat more than 64x above its median, and showed that targeted embedding pruning cut that P95 by over 4x without hurting retrieval quality. Encode and prefill stages over comparable workloads held p99-to-p50 gaps of only 50 to 60ms in 2024 measurements. A 2024 analysis of HNSW's graph structure helps explain the skew: query efficiency leans on hub nodes, so filtered or sparse-region queries can land far outside the typical distribution. Which queries those are depends entirely on your traffic. Hence the baseline.
Segment the percentiles by query type while you're at it. A single p95 across factual lookups and long synthesis requests averages two different populations back into the blur you just escaped.
And treat every vendor number as a hypothesis. NVIDIA's TensorRT-LLM performance documentation warns outright that its published throughput and TTFT figures reflect default configurations rather than peak deliverable performance. Our own experience matches: hitting a sub-200ms time-to-first-token target on a production Rust RAG stack meant re-running every benchmark on our own hardware and traffic, because none of the published figures survived contact with either.
Your baseline is done when each stage has a stable p50, p95, and p99 you'd bet on.
Step 3: Allocate the Budget Across Embedding, Retrieval, Reranking, and Generation
Step 3 converts your baseline into an allocation: pick the end-to-end p95 target from the user-tolerance thresholds, then split it across stages in proportion to what you actually measured, with explicit headroom left over. The split is the deliverable. Every stage owner now has a number they either meet or overspend, and "the pipeline feels slow" stops being anyone's bug report.
Start at the top line. For conversational use, budget near 1 second to first visible token. For search-style tasks, a few seconds of total p95 is defensible, provided something visible happens early. Then divide by measured share instead of reflex. The reflex says generation dominates, and the 2024 systems measurements putting retrieval at roughly 35% of time-to-first-token say the reflex is frequently wrong.
The split below comes from a production pipeline we instrumented during a recent RAG build for a document-heavy enterprise assistant: Milvus vector search, Jina embeddings, a 3-second end-to-end p95 target.
Stage | p95 allocation | Why this share |
|---|---|---|
Query embedding | 50 ms | Cheapest stage in nearly every 2024 measurement |
Vector search + filters | 700 ms | ~35% of TTFT in 2024 studies, and the widest tail |
Reranking | 250 ms | Scales with candidate count k |
Prefill to first token | 900 ms | Scales with retrieved context length |
Decode + render | 1,000 ms | Scales with output length |
Headroom | 100 ms | Network, prompt assembly, surprises |
Two allocation rules earn their keep. Give the widest-tailed stage the most generous share or fix it before budgeting, because a stage whose p95 sits 64x above its median will torch any allocation built on its median. And treat the budget as spendable per query rather than fixed per config.
The SAGE adaptive-retrieval paper from August 2026 is the proof of that second rule. Operating under a 5-second P95 SLO, a learned policy that varied retrieved passages per query across k values from 2 to 30 lifted SLO compliance from 30% to 95%, cut P95 latency from 5.6 to 3.6 seconds, and reduced retrieval cost by 51%. The price was 2 percentage points of exact-match accuracy. A static k of 20 met the same SLO less than a third of the time.
Step 4: Optimize Only the Stages That Overspend Their Budget
Step 4 matches each fix to the stage that's overspending its allocation, because every stage answers to different levers and effort applied elsewhere is effort burned. This is where the measurement pays off. You now know which line item is over budget, and the remediation menu is stage-specific.
One piece of theory sharpens the menu. A 2024 survey framing LLM inference through a roofline model shows prefill is compute-bound while decode is memory-bandwidth-bound, which is why context trimming helps prefill, speculative decoding helps decode, and confusing the two produces optimizations that measure as noise.
Overspending stage | What works | Measured effect |
|---|---|---|
Perceived latency | Streaming | Cuts perceived wait "to a second or less" per OpenAI's guidance, checked 2026 |
Decode | Output-length control | 50% fewer output tokens cuts roughly 50% of generation latency (2026 guidance) |
Decode | Speculative decoding | 2 to 3x decode-latency reductions in 2024 to 2025 studies, workload-dependent |
Prefill | Prompt caching | Up to 80% latency and 90% cost reduction on prefix-shared requests (2026 docs) |
Retrieval | Pipelining with generation | PipeRAG: up to 2.6x end-to-end speedup (KDD 2025) |
Retrieval | Lookahead prefetching | TeleRAG, 2025: 1.53x average latency cut, 1.83x batched throughput |
Retrieval | GPU resource partitioning | Up to 1.5x SLO-compliant throughput without new hardware (2025) |
Reranking | Shrink candidate count k | Jointly comparing 16 candidates ran ~1.75x faster than bi-encoder scoring of 64 (2024) |
A few of these deserve emphasis. OpenAI's latency-optimization guidance calls streaming the single most effective approach for perceived latency, and adds a number that should reorder most teams' backlogs: halving input tokens improves latency by only 1 to 5% in most cases, while halving output tokens halves generation time. If your decode line is over budget, edit the answer format before you touch the retriever.
For prefill overspend, Anthropic's prompt caching documentation reports latency reductions up to 80% on repeated long-context requests that share a prefix. RAG pipelines resend near-identical system prompts constantly. Few levers are this cheap.
For reranking, k is the lever hiding in plain sight. And a 2026 analysis of reranker families found only a handful of models sit on the size-quality Pareto frontier, so check yours is one of them before paying its latency.
Semantic caching belongs on the menu too, with a condition: measure it per traffic segment. A 2024 study of GPT semantic caching hit 61.6 to 68.8% cache rates on repetitive query workloads, which is transformative when it applies and irrelevant when it doesn't.
Step 5: Enforce the Budget With Alerts and Regression Gates in CI
Step 5 wires the budget into alerts and deploy gates so it survives contact with the next model swap, index rebuild, or prompt edit. A budget nobody enforces decays into a wiki page. The half-life is about one quarter.
Enforcement has three layers:
Per-stage percentile alerts. Alert when any stage's p95 breaches its allocation over a meaningful window, and alert on SLO burn rate for the end-to-end target. Stage-level alerts are the point: an end-to-end alert tells you users are hurting, a stage alert tells you where.
A latency regression suite in CI. Replay a representative query set, including the filtered and sparse-region queries your baseline flagged as tail-prone, against every candidate build. Compare per-stage p95 against the current baseline.
Deploy gates on p95 deltas. Block or flag any deploy that moves a stage's p95 past its budgeted share by more than an agreed tolerance. Latency regressions caught pre-deploy cost minutes. Caught in production, they cost an incident review.
Then re-measure after every change that touches the pipeline. A new embedding model shifts index geometry. A bigger context window shifts prefill. A reranker upgrade shifts the k trade-off. Each of these invalidates part of the baseline, and the 2024 evidence that retrieval tails move independently of medians means a green mean after a change proves nothing.
The cadence matters less than the trigger. Quarterly re-baselines are fine for a stable system. What can't be skipped is the re-measure after a deliberate optimization, because that's the moment second-order effects show up.
Ship the gate before you ship the next optimization.
Why RAG Latency Optimizations Backfire When You Skip the Measurement
Unmeasured RAG latency work backfires because second-order effects downstream routinely cancel or reverse the local gain, and the published failure cases are specific enough to name. Every one of them was avoidable with a per-stage baseline.
The clearest example comes from a production system, from people with no paper to sell. A 2026 industry study from Dell Technologies engineers evaluated multi-query retrieval fusion and reciprocal rank fusion in a deployed enterprise pipeline. Fusion added latency through query rewriting and larger candidate sets, and its recall gains were, in the authors' words, largely neutralized after reranking and truncation. The team paid milliseconds for improvements the pipeline discarded before the user ever saw them.
That pattern repeats across the pipeline:
Rerankers off the Pareto frontier. The 2026 calibration analysis of reranker families found only a small set of models sit on the size-quality frontier, so most reranker choices buy latency without accuracy over a cheaper option. Picking one by reputation instead of measurement means paying full price for nothing.
Semantic caches sized on averages. The 2025 category-aware caching study measured hit rates of 40 to 60% on high-repetition query categories and 5 to 15% on volatile conversational traffic. A cache justified by a blended average can be dead weight for half your users.
Co-located GPUs fighting each other. The 2025 GPU-partitioning work showed a retriever and an LLM sharing a card degrade each other's tail latency, so speeding up one stage in isolation can worsen the other's p99 on the same hardware.
Trimming the wrong side of the prompt. Per OpenAI's 2026 guidance, halving input tokens improves latency by 1 to 5% in most cases. Teams aggressively cutting retrieved context while their real overspend sat in output length were spending accuracy to buy noise.
The common thread is blunt. Each optimization was locally sensible and globally wrong, and only an end-to-end measurement, taken after the change, could tell the difference.
Last Steps
The latency budget for production RAG comes down to five moves run in order: instrument every stage, baseline p50/p95/p99 under real traffic, allocate the end-to-end target by measured shares, fix only the overspending stages, then gate deploys on p95 deltas. None of the five is exotic. The discipline is refusing to reorder them.
Treat the result as a living artifact. Every model swap, index rebuild, or prompt change shifts the proportions, and a budget frozen at last quarter's baseline is a comfortable fiction.
This week, do the smallest version: add spans to one pipeline, let real traffic run through it for a few days, and read the per-stage p95s. One of them will surprise you. Measure first. The optimizations will still be there tomorrow.
About the author
Ignas Vaitukaitis is the founder of AlphaCorp AI, where he builds RAG and agentic document pipelines, and maintains RustyRAG, an open-source Rust RAG stack. Commercial disclosure: the author's company offers RAG development services.
disclosure: commercial-author
reviewed_by: Nico Dudli
