A dense 12B beat a sparse 35B MoE at writing search context

I run semantic search over a few thousand recorded meetings. The single thing that makes it work is not the embedding model or the vector store — it is a paragraph of context an LLM writes for every chunk before I embed it. That enrichment step roughly doubled retrieval quality. It is also the slowest, most expensive part of the pipeline, so this summer I tried to make it cheaper by swapping the model that writes the context.

I expected a clean trade: a faster model, slightly worse text, and a prompt tweak to recover the difference. The faster model — a sparse 35B-parameter mixture-of-experts, Qwen3.6-35B-A3B — was about 1.8× faster than the dense 12B I use now, Gemma-3-12B, and by one measure it was actually more faithful. But its context made search consistently worse, and after five prompt rewrites across two opposite directions, I could not recover the difference. The dense 12B stays.

This is the write-up: what enrichment buys, which prompt and chunk size matter, and why the faster model lost — including the parts where my own analysis was wrong until something else caught it.

One caveat before any number: the head-to-head ablations run on a frozen 79-transcript slice of the corpus, where absolute scores sit roughly 2× higher than production because the index is small and the answers stand out. Only the differences between arms are meaningful there. Where I quote a whole-system before/after, it is on the full index and I say so. And every example in this piece is synthetic or redacted — the corpus is real personal meeting data and none of it appears here.

§The question

The corpus is conversational and messy: thousands of meetings, each a wall of speaker turns. The job is to answer a question by returning not just the right meeting but the right moment in it. Plain vector search is mediocre at this. On the first honest benchmark, it found the right transcript about 54% of the time in the top 10 and the right chunk about 17% of the time. Finding the meeting is tractable; finding the passage is the bottleneck, and it stayed the bottleneck through everything that followed.

The fix that changed the system is contextual enrichment: before embedding a chunk, show an LLM the whole meeting plus that chunk and have it write a short paragraph situating the chunk in the meeting. Embed the paragraph-plus-chunk instead of the raw chunk. This is a known idea; the point here is not that it works but how much it dominates everything else, and what happens when you try to make it cheaper.

Because enrichment runs an LLM over every chunk of every meeting, it is the expensive step — the whole corpus rebuild is measured in GPU-hours, not minutes. So the obvious optimization is to write those paragraphs with a faster model. I went in with three beliefs: that once you have context, its exact wording is a big lever; that a bigger, faster mixture-of-experts model would be a free speedup for the write path; and that if a faster model wrote slightly worse context, prompt engineering would close the gap. The first two turned out mostly wrong, and the third I have not been able to make true.

The stakes are ordinary but real: this index backs a production search tool I use daily. A wrong “the faster model is fine” ships a quietly worse index, and nobody notices until search feels off weeks later.

§What the enriched text looks like

The corpus is private, so everything in this section is a synthetic chunk I wrote to mirror the patterns — not real meeting text, not real model output. Take a short exchange:

A: The migration’s basically done, we’re waiting on the DNS cutover. B: What’s the rollback if it goes sideways? A: Keep the old cluster warm for 48 hours, flip back at the load balancer. B: Cost of keeping both up? A: About two grand for the two days. Cheap insurance.

Embedded raw, that chunk is hard to find — it never says “migration plan,” “rollback strategy,” or “infrastructure cost” in as many words. Enrichment prepends a paragraph that does:

A tells B the infrastructure migration is complete except for the DNS cutover. On rollback, A will keep the old cluster warm for 48 hours and revert at the load balancer; the cost is about $2,000 for the two days. Part of the meeting’s migration go-live and risk discussion.

Now a query like “how much to keep both clusters running during the cutover” has something to match. That is the entire mechanism, and it is worth more than everything else in this post combined.

§Setup

Corpus and chunks. ~2,876 meeting transcripts in SQLite with sqlite-vec for vectors. Each meeting is split into overlapping conversation windows — W speaker turns per window, sliding by S turns. Production settled on W=3, S=1 (I’ll come back to why).

Enrichment. For each chunk, the context generator receives the full document and the chunk, and returns a paragraph, which is prepended to the chunk before embedding. Context generation runs on vLLM with prefix caching so the shared document isn’t re-encoded per chunk. The production generator is Gemma-3-12B (GPTQ). Embeddings are Qwen3-Embedding-8B at 4096 dimensions.

The benchmark. 181 hand-checked queries over 79 transcripts, with ground truth at two levels: which transcript is correct, and which chunk is correct. Metrics are recall@k, MRR, and nDCG@10 at both the transcript level (T-) and chunk level (C-). Queries are typed — factual, abstract, temporal, cross-meeting, person-filtered — because the types behave differently and an aggregate hides that.

One benchmark detail matters more than it sounds. Chunk-level ground truth is matched to candidate chunks by time overlap, not by chunk ID. An earlier version matched IDs, which silently rigged window-size experiments: the ground-truth chunks had been created with one window size, so that window size “won” by construction. Time-overlap matching removed the bias and reversed a conclusion (more below).

Statistics. Comparisons use a paired bootstrap (10,000 resamples, fixed seed) for confidence intervals. The model comparison went further: a transcript-clustered bootstrap — resampling whole transcripts, not individual queries, because queries from the same meeting are not independent — and a non-inferiority test with margins fixed before looking at results (−0.03 on T-MRR, −0.04 on C-MRR). Non-inferiority is the honest frame for a swap: a new model isn’t adopted for being “close,” it has to be provably no-worse-than by more than a small margin.

How this was run (the short version). I did not run these experiments by hand. They were executed by an autonomous coding-agent loop: an external nudge fires on a timer, the agent reads a task index and a fixed development SOP, does the work through sub-agents, and stops when the task closes. The first-generation trigger is a shell script that re-sends a prompt file every hour —

# ping_keepalive.sh — types a "continue" message every INTERVAL seconds,
# re-reading its payload from prompt.txt (which another agent can rewrite live)
INTERVAL=3600  # 60 minutes

— and the payload is a decision tree (“read the task index; pick the lowest-numbered incomplete task; follow the SOP; benchmark; run tests; if all done, write a summary and stop”). The SOP forces a specific shape: the main thread only orchestrates, all code goes through a Codex sub-agent, and a second model audits in read-only mode before anything is believed. That audit step is not decoration; it is the reason the numbers below are trustworthy, and I’ll return to it. The rest of the loop machinery — durable docs, memory across context compactions, cooperative GPU leasing negotiated with a separate scheduler agent — is a story for another note.

§The run

I’ll tell this in the order it happened, because the mistakes are part of the result.

Enrichment is the whole game. Adding context, versus embedding raw chunks, was the single biggest jump in the project — on the full index, recall@5 went from 0.318 to 0.591 and MRR from 0.224 to 0.424. Nothing else came close.

The first number was inflated, and I believed it for a day. The very first enrichment measurement looked even better: +106% recall@5, +133% MRR. It was wrong. Only 12 of 2,875 transcripts had been enriched at the time, so the enriched chunks stood out against a sea of un-enriched ones — the benchmark was rewarding contrast, not context. A peer-review audit flagged the mixed-index confound, forced a full re-embed of all 2,876 transcripts, and the honest number came back lower. This is the first of several times the loop’s own first answer was too good and something had to catch it.

The gap that wouldn’t close. Even with enrichment, the transcript-to-chunk gap persisted: the system finds the right meeting far more often than the right moment inside it. Every experiment after this is, in some sense, an attempt to close that gap.

Which prompt? If context is the lever, maybe the wording of the context is a second lever. I ran two ablations. The first tried six literature-backed prompt styles — a generic “situate this chunk,” a rigid metadata format, a speaker-resolved rewrite, a temporal anchor, a question-generation prompt, and a dense “kitchen-sink” line. The generic baseline had the best chunk-MRR; the temporal-anchor prompt was the only one significantly worse; the structured/metadata prompts actually hurt chunk precision, because stamping every chunk of a meeting with the same title/date/participants makes them look alike and harder to tell apart. The lesson was deflating in a useful way: prompt form was past diminishing returns. The existence of context mattered far more than its shape.

The styles are easy to see on that same synthetic chunk (again, illustrative, not real output):

• One sentence: A confirms the migration is done pending DNS cutover, with a 48-hour warm rollback costing about $2,000.
• Structured metadata: Meeting: infra sync. Participants: A, B. Topic: migration cutover and rollback. — but every chunk of the meeting gets the same header, so the chunks stop looking different from one another.
• Detailed paragraph: the one from earlier — who said what, the 48 hours, the $2,000, the load-balancer revert.

The detailed paragraph keeps the specifics a searcher would actually type; the metadata format throws them away and stamps a uniform label instead. That is exactly why structured prompts helped find the right meeting but hurt finding the right chunk.

The second ablation held that lesson but varied length: a terse one-liner (~106 chars), a one-sentence summary (157), a chunk-specific summary (378), and a detailed paragraph (766). Length turned out to buy chunk precision — chunk-MRR rose from 0.295 at ~106 chars to 0.353 at ~766 chars. The detailed paragraph had the best chunk-MRR (+11.4% over the production prompt), though the improvement was not significant at the 95% level (p≈0.035, one-tailed). I’d call it suggestive, not proven — and it becomes important later, because “longer context helps find the passage” is exactly the intuition the model experiment would test to destruction.

Which window? Window size is the other structural knob. Here I got burned by my own benchmark. An early experiment declared the production 5-turn window the winner on all metrics. It was an artifact: the chunk ground truth had been built with 5-turn windows, so exact-ID matching favored them. Rebuilt with time-overlap matching, the result reversed — 3-turn windows significantly beat 5-turn on ranking (T-MRR +6.6%, p=0.012). Smaller windows rank better; wider windows recall better; there is no free lunch, and the earlier “5-turn wins everything” had been a measurement bug wearing a conclusion’s clothes.

The two winners combined — 3-turn windows plus the detailed-paragraph prompt — moved the production config by +10.0% T-MRR and +25.9% C-MRR over the old defaults. That is the configuration the speed question was then asked about.

Can a faster model write the context? Context generation had already come a long way on speed — from 0.27 chunks/sec on a naive setup to 37.78 chunks/sec once it moved to vLLM with a 12B model and prefix caching. But it’s still the pipeline’s slow step, and a sparse mixture-of-experts model is an appealing next lever: Qwen3.6-35B-A3B has 35B parameters but activates only ~3B per token, so it runs fast. Measured on the same prompt, it enriched at 10.72 chunks/sec versus the dense 12B’s 5.84 — about 1.8× faster. A separate dense 27B (Qwen3.8-27B) I also tried was ~2× slower, so it was out on speed alone.

Then I embedded the MoE’s context and scored it. It lost. Not dramatically, but consistently, on every query type, and it failed the non-inferiority test on both metrics. There was a genuine consolation: on a blinded 30-chunk faithfulness check, the MoE actually hallucinated less than the dense model (2 flagged vs 4). Its paragraphs read fine — arguably more careful. They were simply worse at making chunks findable. Faithfulness and retrievability turned out to be different axes.

The difference is visible in the text. Same synthetic chunk, same prompt — the dense 12B writes concretely:

…A will keep the old cluster warm for 48 hours and revert at the load balancer; the cost is about $2,000 for the two days.

and the MoE writes like an analyst:

The team weighs the operational trade-offs of a warm-standby rollback against its incremental cost, reflecting a pragmatic approach to release risk.

(Both illustrative, not real outputs — but faithful to the measured pattern: across the corpus the MoE used abstract framing far more often and kept noticeably fewer concrete numbers.) The MoE version is arguably the nicer read, and that is the problem. The handles a query grabs onto — “$2,000,” “48 hours,” “load balancer” — are softened into “incremental cost” and “operational trade-offs.” Faithful, fluent, and harder to retrieve.

The mess in that experiment. This is where the loop’s first answers were repeatedly wrong. The MoE’s own context regeneration came back 65% empty on one run, traced to a GPU-lease bug where an ID-less keep-alive let the lease silently expire mid-run and the endpoint got swapped out from under the job. One audit model caught a filter that had silently dropped 24 of the 181 queries. Another caught a bootstrap that double-counted queries from multi-transcript questions, which had shifted the confidence interval. Each of those, uncaught, would have changed the published verdict. None was caught by the agent reporting on itself.

Can prompt engineering rescue the MoE? This was the belief I most wanted to be true, because it would mean a free 1.8× on the expensive step. My read was that the MoE wrote in a more abstract register and that retrieval rewards concreteness, so I wrote candidates to push it concrete: preserve names and numbers verbatim, drop the abstract “broader context” language, ban editorialising. All three lost — worse than the MoE on the dense model’s own prompt. So I tried the opposite direction: longer, richer, more relational context. Those lost too. Five candidates, two opposite directions, none non-inferior, none even beating the MoE’s own baseline prompt.

What the candidates revealed was a trade-off with no winning setting. Push the context terser and it loses the thematic framing that abstract, cross-meeting, and person-filtered queries lean on. Push it longer and it finds the right meeting better than any other arm — the 1,460-character candidate had the best transcript-MRR of all — but smears the exact passage, giving it the worst chunk-MRR. The best combined point sat right around 700 characters, which is roughly where the dense model already writes, unprompted.

Both series are point estimates, and closer to zero is better (zero is the dense 12B). The meeting line climbs with length; the passage line peaks in the middle and drops at the long end. There is no character count where both are near zero — and every one of these arms still failed the non-inferiority test on its confidence interval. I stopped before automating the prompt search: five hand-built candidates already spanned the length-and-style space it would explore, and all of them failed.

§Results

All ablation figures are on the frozen 79-transcript index; absolute scores run ~2× production, so read the deltas, not the levels.

Dense 12B vs. sparse 35B MoE (same prompt).

T-MRR delta −0.028 (clustered 95% CI [−0.060, +0.005]); C-MRR delta −0.037 (CI [−0.080, +0.002]). Faster and more faithful; still not adoptable.

The prompt rescue (five candidates, Δ vs. the dense 12B). None clears the −0.03 / −0.04 non-inferiority margin; none beats the MoE’s own baseline prompt (row 1).

The last row is the trade-off in one line: the longest candidate is the best at finding the meeting and the worst at finding the passage.

Prompt length vs. chunk precision (the earlier prompt ablation). Monotone but small; the top-scoring row is not significant at 95%.

Bugs the checks caught before I published a wrong number. Not a footnote — arguably the main result.

§What it means

Four things I’ll claim, in decreasing confidence.

1. Contextual enrichment is the dominant lever for this kind of conversational retrieval — bigger than the embedding model, the window size, or the prompt. If you do one thing, do this.

2. The prompt’s form is mostly a wash; its length is a small, real dial. Rigid metadata formats hurt because they make a meeting’s chunks look alike. Longer, detailed context buys some passage precision, up to a point — but the effect is modest and, in my data, not significant at 95%. Don’t spend weeks on prompt wording.

3. Window size trades ranking against recall, and — the part that should make you nervous — I only learned the true direction after fixing a benchmark that had been quietly grading on a curve. Evaluate chunks by time overlap, at both the transcript and passage level, or you will confirm whatever your ground truth was built with.

4. For writing this search context, the dense 12B beat the faster sparse 35B MoE, and I could not prompt around it. I want to be precise about the scope, because it’s the surprising one. It is a finding on one task (retrieval-context generation), one corpus (meeting transcripts), one model pair, evaluated with the specific candidate prompts I wrote. It is not a law about mixture-of-experts models. What it says concretely: a model’s active-parameter budget and writing style can matter more than its nominal size for this task, and “it’s faster and more faithful” does not imply “it’s better for retrieval.”

Here is the trade-off stated the way it actually bites. Adopting the MoE would buy about 1.8× faster enrichment — a speedup on an offline, batch step that no user ever waits on. The cost is paid on every search: on the benchmark index, roughly −4% on finding the right meeting (T-MRR) and −8% to −13% on finding the right passage (C-MRR and chunk-recall@10). It’s worst exactly where the system is already weakest — abstract, cross-meeting, and person-filtered queries. So the trade is: speed up a background job you don’t feel, in exchange for a quality tax you feel constantly, concentrated on your hardest queries. Framed that way it isn’t close. What keeps it open at all is that I only tried five prompts; “no prompt can fix this” is not what I’m claiming — “the five I tried, spanning terse to verbose, didn’t, and they revealed a trade-off with no obvious sweet spot” is.

And the thing I’d underline for anyone running experiments through an agent: the trustworthy part of this whole account is not the agent’s judgment, it’s the scaffolding around it. Several of the verdicts above were wrong on the first pass and became right only after a second model audited the analysis or a pre-registered gate refused a partial result. An identical loop without the audit layer and the non-inferiority discipline would have shipped several of them. The structure is doing the work.

§Steal this

The reusable recipe, in order of impact:

1. Enrich chunks with whole-document-conditioned context before embedding. This is the win. Everything else is a rounding error next to it.
2. Prefer a detailed enrichment prompt to a terse or rigidly-structured one if passage-level precision matters. Avoid metadata templates that stamp every chunk of a document identically — they collapse within-document distinctiveness.
3. Tune window size, and expect a ranking-vs-recall trade. Smaller windows rank better; wider windows recall better.
4. Evaluate at both the transcript and the chunk level, and match chunks by time overlap, not ID. If your ground-truth chunks were built with one chunking scheme, ID matching will hand that scheme a rigged win.
5. Pre-register non-inferiority margins for any swap. It makes “keep the incumbent” a real, reportable outcome instead of a disappointment, and it stops “close enough” from becoming “ship it.”
6. Regenerate baselines in the same session as the challenger. Copied production vectors carry their own drift (I measured cosine 0.925 vs 0.973 between copied and freshly-regenerated baselines) and will quietly bias the comparison.
7. Have a different model audit the analysis, read-only. Not to write code — to try to break the conclusion. In this project that step caught more real errors than any other single practice.
8. When you evaluate a faster model for a generation step, gate on the downstream task, not on the step’s own metrics. The MoE was faster and more faithful and still wrong for the job. Speed and faithfulness are not retrieval quality.

The negative result is the useful one here: a faster, larger-on-paper model was available, and the disciplined answer was to not adopt it — and to be able to say exactly how much it would have cost if I had.