Agent Memory Atlas

Pattern · Retrieval

Hybrid Retrieval Fusion

Combine semantic similarity, lexical exactness, metadata filters, and bounded reranking instead of trusting one retrieval signal.

Epistemic status Reporting an established practice

Intent

Retrieve both conceptually related memory and exact facts, while enforcing scope and lifecycle constraints before results reach the agent.

The problem

Vector search is good at paraphrase and topical similarity but weak on exact names, identifiers, dates, paths, and negation. Lexical search catches exact strings but misses paraphrase. Either signal alone produces avoidable blind spots.

The size of that blind spot is measurable. On LIMIT, a benchmark built to expose single-vector retrieval, BM25 reaches Recall@20 of 0.9490 where Contriever reaches 0.0265 (Clavié et al., Latent Terms, arXiv:2605.29384, 28 May 2026). That paper also shows some of the gap is recoverable from the dense model — but recovering it is a second index, not a better read of the first. It trains a new sparse autoencoder on the retriever's frozen final-layer token activations and scores the resulting features with BM25 over an inverted index; the pooled embedding is never rescored. The recovery is partial (Contriever 0.0265 → 0.5100, still 0.44 short of BM25) and it does not hold everywhere — on a multi-vector model it loses ground on both LIMIT and BEIR. Read it as evidence that lexical structure exists in token-level activations and needs its own scoring machinery to be usable, which is an argument for keeping a lexical arm rather than for expecting an embedding to grow one.

The pattern

Apply hard filters first, retrieve candidates through independent channels, normalize or rank their outputs, then fuse and cap the final context:

Diagram — hard filters run before any candidate is scored, three arms then produce candidates that are fused, optionally reranked, and cut to a token budget
Diagram source
%% caption: hard filters run before any candidate is scored, three arms then produce candidates that are fused, optionally reranked, and cut to a token budget
flowchart TD
    Q["Query"] --> F["Hard scope, lifecycle,<br/>sensitivity filters"]
    F --> V["Vector<br/>candidates"]
    F --> L["BM25 / FTS<br/>candidates"]
    F --> E["Entity or<br/>exact-key candidates"]
    V --> R["Rank fusion"]
    L --> R
    E --> R
    R --> X["Optional reranker"]
    X --> B["Diversity +<br/>token budget"]

Reciprocal-rank fusion is a useful baseline because it combines rankings without pretending incomparable scores share a calibrated scale — the property its authors claim for it, "without regard to the arbitrary scores returned by particular ranking methods" (Cormack, Clarke and Büttcher, SIGIR '09). Weighted score fusion can work when every component has measured, bounded behavior.

The constant is a default, not a measurement. k = 60 was fixed in a pilot that fused thirty configurations of a single search engine, over a curve flat from 30 to 100; the paper's own words are that 60 "was near-optimal, but that the choice was not critical", and 80 scored fractionally higher in the table it comes from.

What k controls is how much of an arm's rank ordering survives fusion. Within one arm, k = 60 compresses the whole top 60 into a factor of 1.967× — so a first-place hit is worth barely more than a sixtieth-place one, and cross-arm agreement dominates rank position. How much agreement is worth is not a fixed factor: a document at rank a in one arm gains 1 + (k+a)/(k+b) by also appearing at rank b in the other, which reaches 2× only when the two ranks coincide and falls to 1.06× for a = 1, b = 1000. Two more things the arithmetic alone will not tell you: how deep each arm's candidate list runs, since a document missing from an arm's top-k has no defined rank and is usually approximated over the union of the lists; and how correlated the arms are, since agreement between two views of the same corpus is much weaker evidence than agreement among many independently built systems.

Fusing rank-only also discards the score distribution, and the literature has measured what that costs. Bruch, Gai and Ingber, An Analysis of Fusion Functions for Hybrid Retrieval, TOIS 2023 (arXiv:2210.11934) sweeps a separate constant per arm over 1–100 across nine datasets, and finds NDCG "swings wildly" with them. Their reframing is worth the detour: allow one constant per arm and RRF has m parameters against a convex combination's m−1, so the fusion usually chosen for being parameter-free has one parameter more than the one it is chosen over. Three results worth carrying:

  • A convex combination of normalized scores, with one tuned weight, beat RRF (60,60) on NDCG on all nine datasets, in-domain and zero-shot, and needs only a small tuning sample. Normalization choice barely mattered.
  • Symmetrically lowering the constant transferred. RRF (5,5) matched or beat (60,60) on all nine.
  • Asymmetric tuning did not transfer. (10,4), tuned in-domain, improved MS MARCO 0.425 → 0.451 and then cost HotpotQA 0.675 → 0.621 and FEVER 0.721 → 0.649. In-domain the tuning discounts the lexical arm; out-of-domain that reverses.

Akarsu et al. (arXiv:2604.01733), 2 April 2026, agrees in one domain on two arms: convex combination at α = 0.5 reaches Recall@5 0.726, RRF k = 10 0.716, RRF k = 60 0.695.

So: keep 60 if you have nothing to tune against, and do not defend it as measured. If you can normalize scores and tune one weight on a small in-domain sample, that is the better-evidenced fusion. And if you tune per-arm constants, tune them on your own traffic and expect them not to survive a domain change.

Why it works

The channels fail differently. Semantic search recalls meaning; lexical search protects exactness; metadata enforces boundaries; entity or key lookup handles structured identity. A reranker can resolve close candidates after cheaper retrieval has narrowed the set.

Tradeoffs

  • More channels add latency and operational complexity.
  • Ad hoc score normalization makes rankings hard to explain.
  • Heuristic bonuses can dominate if they are not bounded.
  • Reranking without evaluation adds cost without known benefit.
  • The final result and token limit must be enforced after fusion.

Do not call retrieval “hybrid” merely because two backends exist. The important behavior is intentional candidate fusion under one measured cutoff.

Cost to adopt

Build: a second retrieval arm, a fusion step, and a way to inspect which arm contributed each result. RRF is a dozen lines; the inspection is what makes it maintainable.

Forces elsewhere: each arm needs its own index kept in sync with writes, and a backend that silently lacks one of them degrades to single-arm search while still reporting itself as hybrid — a failure this atlas has found more than once.

Ongoing: fusion weights are tuning surface, and nothing tells you they have drifted except a benchmark you probably do not have.

Skip it if your corpus is small enough that exact search alone answers well. Boring FTS over a few thousand records outperforms an untuned hybrid stack.

Seen in the atlas

Gini is a legible implementation, because it documents its channels and their provenance in a header comment: semantic cosine, BM25 over FTS5, graph spreading activation seeded from the top semantic hits with decay δ=0.5, and a temporal range match — fused with reciprocal rank fusion and reranked. The temporal arm only participates when the query contains a temporal expression, which is a small idea worth copying: a channel that cannot contribute should not dilute the fusion.

HippoRAG is the instructive alternative — it does not fuse at all. It seeds a personalization vector from query-linked entities plus a weak dense prior (passage nodes at 0.05) and reads relevance off a Personalized PageRank diffusion. Multi-hop association becomes a property of the diffusion rather than of a traversal policy or a weighted sum.

LlamaIndex composes instead of fusing: chat history takes a fixed ratio of one token budget and the memory blocks contribute independently into the remainder. The composition has a hole where the budget should be enforced. atruncate is a contract no shipped block implements, the default priority of 0 means never truncate, and a block the orchestrator does truncate is dropped whole, so with default blocks the budget holds only if chat history absorbs the overrun.

Hindsight runs four arms with task-specific fusion and cross-encoder reranking; MemPalace contributes the reusable rule that extracted indexes boost drawer ranking but never gate direct evidence retrieval; mem0, Basic Memory, agentmemory and Magic Context all fuse lexical and semantic signals. CowAgent has both arms over its chunk index, and how they are combined is not traced in its report. OpenViking embeds each query once into dense and optional sparse vectors; its report does not call the sparse side lexical. Honcho keeps the arms apart: message search is ILIKE over the content or an embedding query, never both in one ranking.

Two cautions have strengthened with evidence. Naming is not fusing — Claude-Mem's ordinary text search selects semantic rather than combining it with FTS, and A-MEM's "hybrid" path is vector-only. And silent degradation is worse than no fusion: Holographic redistributes its weights to lexical-only when NumPy is missing while is_available() still returns True, where Moltis makes the same situation explicit with a keyword_only() constructor and a has_embeddings() predicate callers can branch on.

Cortexes adds a third caution: a filter is only as strict as its weakest arm. Its vector stream hands the --repo clause to Chroma, while BM25 and the wikilink graph evaluate it in Python, and until cortex-vec 0.9.0 the nested $or was ignored by one arm and never passed to the other. The repair keeps field names and comparisons in one table that raises on anything unmodelled, and a test fails when the clause producer emits a field that table lacks. Its degradation is explicit in code where Holographic's is not — without a key the vector stream is skipped and the weights renormalized over the streams that answered — but not in the result: the cosine score the agent reads is 0.0 alike for no key, a failed backfill and a real zero. The same key gates writes to the lexical index.

Few systems defend their weights. MetaClaw replays candidate policies against past turns and promotes one only on non-regression across eight metrics. Somnigraph sets a separate RRF constant per arm — about eight for FTS5, about seven for vectors — and a vector weight by a named tuning study, each constant carrying its previous value in a comment. Uteke tunes its fusion weights on a LongMemEval subset and writes the tuning set and plateau beside the constant. Most others, Generative Agents with its hand-tuned gw = [0.5, 3, 2] among them, ship constants nobody has defended, most often RRF's k = 60.

Helm is a small correct instance — both arms and the fusion are about 120 lines of JavaScript over rows already in memory, with no FTS extension and no vector store — and it is a clear place to see that fusion quality is bounded by candidate generation, not by the fusion rule. Its RRF is textbook (k=60, no score normalization, which is the right refusal when you have no relevance data to calibrate against), and its belief weight is applied as a multiplier rather than a filter so a low-confidence row is penalized instead of excluded. Then every arm runs over SELECT … WHERE expired_at IS NULL ORDER BY updated DESC, confidence DESC LIMIT 500 — a hard window ordered by recency. Past a few hundred active facts, the memories that stop being candidates are the ones nothing has touched lately, which is precisely the set of long-lived preferences the store worked hardest to establish. A perfect ranker over the wrong 500 rows is still wrong, and no amount of fusion tuning recovers a candidate that was never scored.

Helm is also a plain example of silent tier degradation. The semantic arm is a cached MiniLM embedding if the model is on disk, and TF-IDF cosine if it is not or if the import throws, because the TF-IDF scores are computed first and an empty catch {} keeps them; with no model and fewer than three active facts the arm is all zeros. Three materially different retrieval qualities sit behind one output shape, with no signal to the caller. If a channel can degrade, the result should say which channel ran.

Qwen MM Plugins closes a failure mode this page has otherwise only described in its degradation form: the two arms can also disagree about what the vectors mean. Its dense arm reads an embeddings.npz written at build time, its sparse arm is a BM25 index built over the same nodes, and both are fused by RRF in the usual way. What is uncommon is check_dimension_compatibility, which compares the stored matrix's width against the embedding backend now answering queries and raises with a message naming the likely cause — a memory "built with a different model than the current query backend". A system that persists a dense index and embeds queries at read time holds two artifacts that can drift apart with no exception, no crash and no empty result: the dense arm keeps returning neighbours, they are simply neighbours in a space the query never entered, and RRF dutifully fuses that garbage with a sparse arm that is working correctly. Fusion makes this harder to notice than a single-arm retriever would, because the lexical hits keep the result list looking plausible. A width comparison at load is a one-line check against the highest-cost silent failure in the pattern. Qwen's loader catches the error, disables the dense arm and logs the cause, so the mismatched vectors never rank anything; its search_nodes result carries no field saying BM25 answered alone, which leaves it in the degraded-channel case above.

The same system also shows why fusion earns its keep on content nobody wrote prose for. Its two arms are searched over a video's extracted events, and the lexical side reaches material the dense side systematically blurs — a name, a score, a caption, a slide title, captured verbatim into an OnScreenText record. The argument for hybrid here is not that one arm is better tuned; it is that proper nouns and short strings on screen have no useful embedding neighbourhood at all.

Muninn contributes the placement question this page has not asked: where does the access filter go? Its fusion is one Postgres statement — an FTS CTE ranked by ts_rank, a vector CTE ranked by embedding <=> $3, each LIMIT 30, joined FULL OUTER on id and scored 1.0/(60 + f.rank) + 1.0/(60 + v.rank) — and the scope predicate bot_name = $5 AND ((scope = 'personal' AND user_id = $1) OR scope = 'shared') is written into both CTEs rather than wrapped around the join. On an instance that authenticates the predicate becomes bot_name = $5 AND user_id = $1, and the no-embedding fallback carries the same branch, because a fix applied to one search function is inert on the path that takes the other.

The difference is not correctness, it is recall, and it falls on the caller with the fewest permissions. Filter after the fusion and each arm still spends its 30 candidate slots on whatever ranks highest globally, including rows the caller may not see; the survivors are whatever is left. Filter inside each arm and both budgets are spent within the boundary. A reader comparing two systems cannot see this in a benchmark — the post-filtered version returns fewer, worse results only for the narrowest-scoped users, which is exactly the population least likely to be in the golden set.

Muninn also shows the degradation worth having: the vector CTE carries embedding IS NOT NULL, so a row saved when the embedding model was unavailable is excluded from that arm and stays reachable through FTS, with getMemoriesWithoutEmbeddings available to backfill it. One arm failing costs one arm, not the query.

repowise supplies the failure this page's degradation advice exists to prevent, measured. Its vector leg was bounded at a hardcoded 8-second timeout inside contextlib.suppress, while a cold index costs "6.3s + 13.4s ... where a warm query takes 0.19s" — so, in the fix's own words, "the first query of every process expired, the leg returned [], and search silently degraded to full-text with nothing logged and embedder_degraded still false."

Three things compose into an unobservable outage, and each is worth checking separately in any fused retriever. The budget was set from warm-path timings, so it could never clear a cold start. The suppression made the expiry indistinguishable from an empty result set — a fused query cannot tell "the vector arm found nothing" from "the vector arm never ran". And the health flag that exists to report exactly this stayed false throughout, so the one field an operator would query to detect the degradation reported health. The remedy is the shape to copy as much as the number: one shared budget rather than a constant at each call site, an override for slow disks, and an unusable override that warns and keeps the default "rather than disabling the leg".

Test that an arm's failure is distinguishable from an arm's empty result. No fused implementation on this page has that fixture, and it is the difference between a retriever that degrades and one that degrades silently.

Forgetful is the counterexample to keep beside these. Its README, the four-stage docstring on search in both of its repositories, and the recall skill the agent loads all describe dense → sparse → reciprocal rank fusion → cross-encoder, and the tree implements dense → cross-encoder: no FTS table, no tsvector, no fusion function. The identifiers the skill says are matched "literally" are matched by cosine over a 384-dimension embedding. A fusion that is documented and not built is worse than one never mentioned, because the query-shaping advice — put the exact error code and config key in the query — is written for the arm that is missing.

Uteke tunes the fusion and then leaves its scale. The default recall fuses two complete rankings — vector at weight 1.7 and a vector-plus-FTS5 RRF at 1.0, k = 60 — with the tuning set and plateau written in the constant's comment, and the committed LongMemEval-S raw output recomputes the published 98.4% recall_any@5. After fusion a memory's score is the RRF sum, at most 2.7 / 61 ≈ 0.044; the default path then adds 0.1 × salience + 0.1 × recency and applies thresholds written for cosine, so --strict at 0.5, the Hermes hook at 0.40 and the pi hook at 0.45 exclude every memory. RRF's virtue of not pretending incomparable scores share a scale lasts only until something downstream pretends again.

Claude Self-Reflect is the case for fusion, made accidentally by a system that does not fuse. Its FTS5 arm runs only when the best semantic candidate scores below 0.5, and when it runs every hit enters at a constant 0.45 — FTS5's own rank orders the SQL and is then discarded, so BM25 never reaches the score. The project's own pre-registered benchmark built a proper fusion arm for comparison and measured all three: reciprocal-rank fusion over vector top-20 and FTS top-20 scored 0.813 over 396 receipt-lookup queries, vector alone 0.697, and FTS alone 0.162. Lexical exactness was the weakest single channel and worth roughly twelve points in combination — the shape this page predicts, published by a system whose shipped read path leaves those points on the table.

GrayMatter fuses three signals and tests the claim that one of them is a sliding window. RRF over a vector cosine ranking, a TF-IDF keyword score with optional stemming, and recency. Two details are worth copying. Superseded facts are removed before anything is scored rather than filtered from the result, so a retired fact cannot occupy a slot a live one needed. And a decision record asserts that a sliding window is the special case of this ranking with all weight on recency, so the benchmark implements that window independently and asserts on every invocation that the recency-only weighting returns exactly the same fact set — a design claim turned into a check that fails when it stops holding. The published miss is equally instructive: the corpus's one failing query asked about "roll back" against a stored "Rollbacks", which is what motivated stemming, gated on a strict-subset property rather than a net win count.

Graft fuses a title-embedding vector list with BM25 over title and over body by reciprocal rank, and its lexical arms are narrower than they look. build_scoped_fts_query quotes each whitespace token and joins them with a space, which FTS5 reads as AND, so a sentence-length query matches only rows containing every word and the fusion quietly becomes the vector list alone. Heimdall, which vendors the same C, located this join as one cause of its measured recall.

nano-brain shows what a relaxation step does to the predicates around it. Its HybridSearch retries an empty AND lexical leg with a separate OR query that keeps the workspace predicate and has no time-range or tag parameters, so a time-filtered memory_query can fuse out-of-window lexical hits with an in-window vector leg. The same server's memory_search runs the same fallback and refuses it whenever a time filter or tags are set, with the reason in a comment. Build the fallback by changing only the match clause, or test every relaxed path with a filter the first query applied.

Tests to require

  • Exact identifiers, paraphrases, dates, negation, and typo cases.
  • Scope leakage and lifecycle filtering before ranking.
  • Ablations for each retrieval channel.
  • A sweep of the fusion constant on your corpus and arm count, not the paper's — and a check that whatever you pick still holds on data you did not tune against, because per-arm tuning is the setting that has been measured failing to transfer.
  • Hard assertions that @k evaluates exactly the first k results.
  • Token-volume and latency reporting.
  • Stable tie-breaking and bounded heuristic contributions.