Send Your Retriever a Query, Not a Transcript

Most multi-turn RAG chatbots work like this: append each turn to a message list, send the whole thing, let the model sort it out. This is a reasonable default and it mostly works.

But it quietly conflates two different consumers. Your generator wants the conversation — tone, prior corrections, things the user already ruled out. Your retriever wants one clean, self-contained query. Feeding both the same representation means one of them is getting the wrong thing, and it's almost always the retriever.

Splitting them is a small change with a measurable payoff.

A user asks: "Can you share the September financials?"

That string is what gets embedded. Nothing in it says the last four turns were about serverless architecture, or that "financials" here means Lambda spend. Generic question in, generic chunks out — and the failure is silent, because the pipeline returns plausible-looking results for a question nobody asked.

The obvious workaround — concatenate recent turns into the query — helps less than you'd hope. A long blob pulls the query embedding toward the average of everything in it rather than the thing actually being asked, and BM25 picks up terms the user didn't mean. That's the mechanism, not a measured result, but it matches what most teams find when they try it.

The fix is to make one small LLM call that rewrites the query into a standalone form before retrieval: "September financial data for AWS Lambda in the serverless optimization context." Nothing about the history changes; you're just giving the retriever something it can work with.

It's called Conversational Query Reformulation (CQR), and the literature goes back to 2019 — CANARD and QReCC are the standard datasets, TREC's CAsT and now iKAT are the evaluation tracks. LangChain ships it as "condense question."

It's in production at scale. Glean does it explicitly in what they call the Plan step: "we rewrite the query into a multi-step plan" (Glean, Nov 2024).

The research frontier has moved somewhere useful: training the rewriter against the retriever's preferences instead of human-sounding phrasing. ConvSearch-R1 (EMNLP 2025) uses reinforcement learning with retrieval rank as the reward signal and reports 10.3% and 10.7% average gains across metrics on TopiOCQA with 3B backbones, without any external supervised data. Earlier work found automated rewrites could edge out human-written ones on QReCC — on MRR under sparse retrieval, so a narrow result, but pointing the same way.

The transferable lesson, even if you never fine-tune anything: a rewrite that reads well to a human is not the same as a rewrite that retrieves well. Judge it on recall@k, not prose quality.

You've added a probabilistic step upstream of everything. It fails in three ways worth naming.

Topic drift. The user changes subject; the rewriter, primed on ten turns of one topic, drags that context into a question that has nothing to do with it. This is common enough that TopiOCQA exists specifically to benchmark topic switches.

Hallucinated context injection. The rewriter invents an entity, date range, or scope that appears nowhere in the conversation. Retrieval then filters hard on something fictional.

Over-stuffing. A rewrite that packs in too much context recreates the dilution problem you were trying to escape.

None of these have a clean solution. You're asking a model to guess which context is relevant before you know what the user meant, and a wrong guess is indistinguishable from a right one at the moment you make it. What you can do is contain it.

Have the rewriter emit structured output that names its sources, the turn indices and exact spans it drew from. Then verify in code: every entity in the rewrite must string-match a cited span. No match, drop the rewrite and fall back to the raw query.

This doesn't make the rewriter smarter, and it doesn't eliminate fabrication, the model can still pick the wrong spans, or assemble real fragments into a wrong meaning. What it does is make unattested entities mechanically detectable, which converts your worst failure class from silent to caught.

Two things that pair well with it. Fire retrieval on the raw query in parallel while the rewriter runs, if the rewrite comes back similar, you already have results and the extra hop cost you nothing in wall-clock. And keep the JSON schema flat, constrained decoding on deeply nested schemas measurably slows generation on most engines.

This is where generic advice stops being useful.

Mostly one-shot questions? The highest-value addition is a gate, detect that a query is already self-contained and skip the rewrite. Unconditional rewriting is corrupting queries that never needed context.

Mostly follow-ups? That gate is nearly irrelevant, and the rewriter stops being a preprocessing step. It becomes your primary retrieval interface, and its failure rate is roughly your system's failure rate. Your dominant risk also shifts: not topic drift, but error compounding down long chains, where a wrong entity introduced at turn 3 propagates through every rewrite after it. That argues for maintaining an explicit state object — active entities, active filters, current subtopic - rather than re-deriving context from raw history on every turn.

Same technique, different priorities. Anyone selling you a universal RAG architecture hasn't asked what your conversations look like.

There's a related argument that full history bloats the generator's context and degrades reasoning. It's real, but be honest about scale: a twelve-turn chat is a few thousand tokens, and models handle that fine. The attention-budget concern bites on long-running agent sessions and large retrieved payloads, not on a normal support conversation.

Worth knowing, not worth optimizing first. And note that prompt caching — up to 90% cost and 85% latency reduction on cached prefixes — makes keeping full history in the generation call cheap. Cheap on price, anyway; caching discounts the bill, not the attention cost. But at conversational lengths that distinction rarely matters.

The retriever is where the win is.

Log the raw query, the rewritten query, and the retrieved doc IDs across a few hundred real conversations. Compare recall, rewritten versus raw.

If the rewrite isn't beating the raw query on retrieval hit rate, it isn't earning its latency — and no amount of schema design fixes that. Most teams stack techniques without ever checking which ones help.