LlamaIndex

1. Executive Summary

LlamaIndex is a Python framework for retrieval and agents, and its llama-index-core/llama_index/core/memory/ is about 2,330 lines, and it clears this atlas's scope bar where BeeAI does not: FactExtractionMemoryBlock extracts durable facts that survive the conversation, rather than only deciding which messages stay in the window.

The architecture is a single token budget split between two tiers, with flow between them:

Memory(
    token_limit,                      # total budget
    chat_history_token_ratio = 0.7,   # share reserved for short-term history
    token_flush_size,                 # how much spills to long-term at a time
    insert_method = SYSTEM | USER,    # where blocks land in the prompt
)

Short-term chat history gets 70% of the budget by default. When it overflows, token_flush_size worth of messages are flushed into the memory blocks, which own the remaining share. Long-term memory is therefore fed by short-term pressure rather than by a separate capture path — the two tiers are one pipeline, not two systems.

Blocks are the composable part. BaseMemoryBlock is generic over its content type and declares three operations: aget (contribute to context), aput (receive flushed messages), and atruncate(content, tokens_to_truncate), a hook through which a block could shrink itself when the budget is tight. The hook is a contract with no implementation behind it: the base method returns None, which drops the block's whole contribution, none of the three shipped blocks overrides it, and a block's priority defaults to 0, which the orchestrator reads as never truncate. With default blocks, memory content is never cut, and the budget holds only if chat history absorbs the overrun.

Three blocks ship: StaticMemoryBlock (fixed content), VectorMemoryBlock (retrieval over past messages), and FactExtractionMemoryBlock (LLM-extracted facts with a max_facts cap).

The reservation is the familiar framework one: these are primitives, and memory policy is left to the application. There is no trust state, no provenance and no correction path, and the one scope key, session_id, is applied only by the vector block — through a filter the block writes into its own query_kwargs on the first call and keeps for every later one, so a vector block instance reused by a second session goes on retrieving the first session's messages.

2. Mental Model

%% caption: the token budget is split by ratio between chat history and memory blocks, and history overflow is flushed into the blocks rather than dropped
flowchart TB
    TL(["token_limit"]) --- SPLIT{{"split by chat_history_token_ratio<br/>default 0.7"}}
    SPLIT --> CH["chat history<br/><i>0.7 of the budget</i>"]
    SPLIT --> MB["memory blocks<br/><i>the remaining budget, split among them</i>"]
    CH -->|"overflow: flush token_flush_size worth of messages"| MB
    MB --> SB["StaticMemoryBlock<br/><i>fixed content</i>"]
    MB --> VB["VectorMemoryBlock<br/><i>retrieval over past messages</i>"]
    MB --> FB["FactExtractionBlock<br/><i>facts, capped at max_facts</i>"]
    SB --> ASM
    VB --> ASM
    FB --> ASM
    CH --> ASM["assembled prompt<br/>insert_method: SYSTEM or USER"]

The arrow from chat history into the blocks is the design. One budget is divided by ratio, and messages evicted from the conversation half are handed to the blocks rather than discarded — so a block is not a separate store beside the history, it is where the history goes when it no longer fits.

Block interface:

class BaseMemoryBlock(BaseModel, Generic[T]):
    async def aget(...) -> T                 # contribute content
    async def aput(messages) -> None         # receive flushed messages
    async def atruncate(content: T, tokens_to_truncate: int) -> Optional[T]

Returning Optional[T] from atruncate lets a block decline to shrink and be dropped rather than emit something misleading when cut in half — and since no shipped block overrides it, dropping is the only behaviour that ships.

3. Architecture

• memory/memory.py (873 lines) — Memory, BaseMemoryBlock, InsertMethod, budget arithmetic. Memory accepts any AsyncDBChatStore through chat_store, falling back to a SQLAlchemyChatStore built from its own parameters.
• memory/memory_blocks/ — static.py (39), fact.py (182), vector.py (201).
• Legacy surfaces retained: chat_memory_buffer.py (168), chat_summary_memory_buffer.py (340), vector_memory.py (206), simple_composable_memory.py (163).
• memory/types.py (152) — base types.

The legacy modules are the conversation-window family the atlas excludes from scope; the Memory plus blocks API is what makes LlamaIndex reviewable here. Both ship, which is worth knowing when reading the docs — "memory" in LlamaIndex means two quite different things depending on which API you land on.

%% caption: the same budget split seen from the flush side — overflowing messages become static, vector and fact blocks before reassembly
flowchart TD
  Msgs["Chat messages"] --> Hist["short-term history<br/>(0.7 of budget)"]
  Hist -->|overflow: token_flush_size| Flush["flush"]
  Flush --> Static["StaticMemoryBlock"]
  Flush --> Vec["VectorMemoryBlock"]
  Flush --> Fact["FactExtractionMemoryBlock"]
  Static --> Assemble["aget + atruncate<br/>to budget"]
  Vec --> Assemble
  Fact --> Assemble
  Hist --> Assemble
  Assemble --> Prompt["prompt (insert_method:<br/>SYSTEM | USER)"]

4. Essential Implementation Paths

Budget arithmetic with guards

Validation clamps the configuration rather than trusting it: if token_flush_size is unset or exceeds token_limit, it is reset to 10% of the limit; a separate path resets it to 70% if it arrives larger than the limit through the constructor. Defensive, and the kind of thing that prevents a misconfigured flush from evacuating the entire history in one step.

Self-truncating blocks

atruncate(content, tokens_to_truncate) is the right contract and an empty one. When memory blocks plus chat history exceed token_limit, _truncate_memory_blocks walks the blocks sorted by priority ascending, skips every block at 0, and asks each of the rest to give back tokens; whatever is still over is then removed block by block in the same order. The base atruncate returns None, and StaticMemoryBlock, VectorMemoryBlock and FactExtractionMemoryBlock inherit it, so a truncated block loses everything — the vector block does not drop its lowest-scoring node and the fact block does not drop a fact. Two details make the defaults worse than that. priority defaults to 0, so an application that does not set it gets blocks that are never truncated at all. And the field's description — "0 = never truncate, 1 = highest priority" — is the reverse of the order truncation walks, which reaches priority 1 first; aget sorts the other way, by -priority.

Compare how other systems handle an over-budget context. Hermes Agent refuses the write and makes the model consolidate. Voyager concatenates its entire skill library with no budget at all. GenericAgent caps its index at thirty lines by policy. LlamaIndex designs for delegating the how of shrinking to the component holding the content, and ships no component that does it.

Fact extraction and condensation

FactExtractionMemoryBlock runs two prompts. The extract prompt receives existing_facts and is told not to duplicate them, so extraction is incremental. When the count passes max_facts, the condense prompt runs — and its instruction is unusually careful:

"The condensed list you return completely replaces the existing facts, so it must be the full, self-contained snapshot of everything worth keeping — not just newly added or changed facts."

That sentence exists because the obvious failure is a model returning a delta where a replacement was expected, silently deleting everything it did not mention. LlamaIndex addresses it by making the contract explicit in the prompt. ByteRover addresses the same failure by diffing before and after and merging back what would be lost — a guard rather than an instruction. The prompt is cheaper; the diff is verifiable.

The extract prompt also carries an epistemic constraint: "Do not include opinions, summaries, or interpretations — only extract explicit information." That is the same instinct as GenericAgent's action-verified axiom, at much lower strength — an instruction against inference rather than a requirement of evidence.

Insert method

InsertMethod.SYSTEM or USER determines whether block content lands in the system prompt or as a user message. This matters more than it looks: system-prompt placement is stable and cache-friendly (the concern driving Hermes Agent's frozen snapshot), while user-message placement is fresher but breaks the prefix. LlamaIndex exposes the trade as a setting rather than choosing for you.

5. Memory Data Model

There is no memory record. A block owns whatever representation it likes — BaseMemoryBlock is generic over T — so facts are a string, vectors are retrieved nodes, static content is fixed text.

The consequences are the ones the atlas records for every framework primitive layer:

• No status, confidence, or verification. An extracted fact is a line in a list.
• No provenance. A fact does not record the messages it came from, even though those messages were in hand at extraction time.
• No correction. Facts change only by condensation rewriting the whole list.
• Scope only on the vector path. Flushed messages carry session_id into the vector node's metadata, and VectorMemoryBlock._aget adds a session_id MetadataFilter — but only if no session_id filter is already in its query_kwargs, which it has just mutated on the previous call, so the first session's filter sticks to the block instance. FactExtractionMemoryBlock keeps facts as a list on the instance, so a fact block shared by two sessions shares its facts. No test in tests/memory/blocks/ exercises two sessions.
• No per-fact removal and no tombstone. The list changes only through extraction and condensation; flushed messages are archived in the chat store, so a conversation is not re-extracted, but a fact condensed away can be extracted again from a later conversation.

That is a reasonable position for a framework — langmem makes the same choice — but it means "LlamaIndex has memory" and "LlamaIndex has a memory system" are different claims.

6. Retrieval Mechanics

VectorMemoryBlock retrieves over past messages using the framework's vector-store abstractions, so any LlamaIndex-supported backend applies. The query is the last retrieval_context_window messages joined, and each flush becomes one node holding the whole flushed batch. There is no fusion with a lexical arm inside the block, and no reranking in the memory layer — both available elsewhere in the framework, neither wired in by default.

Because blocks compose, hybrid behaviour is achievable by stacking blocks rather than by fusing scores inside one retriever. That is a different composition model from the weighted-fusion approach most of the atlas uses, and it has a real property: each block's contribution to the final context is separately visible and separately budgeted, which makes the assembled prompt easier to reason about than a single fused ranking.

7. Write Mechanics

The only write path into long-term memory is the flush from short-term history. There is no explicit "remember this" surface at the Memory level and no actor model — whatever survives the flush is what the blocks see, and each block decides independently what to keep from it.

Coupling long-term capture to short-term overflow is elegant but has a consequence worth stating: a short conversation never writes long-term memory at all. If history never exceeds its share of the budget, nothing flushes, and the fact block never runs. Memory formation depends on conversation length rather than on content importance.

8. Agent Integration

Memory plugs into LlamaIndex agents and workflows. Custom blocks are the extension point — subclass BaseMemoryBlock, implement three async methods, add it to the list. That is a genuinely low-friction way to add a memory kind.

9. Reliability, Safety, and Trust

Strengths:

• One budget, explicitly split, rather than two subsystems with independent limits.
• A self-truncation contract that would delegate shrinkage to the component that understands the content, if a block implemented it.
• Configuration clamping on flush size.
• An explicit full-replacement contract in the condense prompt.
• An anti-inference instruction in the extract prompt.
• Insert placement as a setting, exposing the cache-versus-freshness trade.
• Composable blocks whose contributions stay separately visible.

Gaps:

• No trust, provenance or correction — application-owned by design.
• No shipped truncation. Blocks default to never-truncate, and a truncated block is dropped whole.
• A session filter that sticks to the block instance, so reusing a vector block across sessions retrieves the first session's messages.
• No tombstone, so a condensed-away fact can return from a later conversation.
• Write depends on conversation length, not importance.
• Condensation is a wholesale rewrite guarded by prompt wording rather than verification.
• Two coexisting "memory" APIs, one of which is conversation-window management, which invites confusion about what the framework actually provides.

10. Tests, Evals, and Benchmarks

llama-index-core/tests/memory/ holds 85 test functions across ten files, three of them for the shipped blocks; nothing was run for this review, and no memory-quality benchmark was found. No test asserts that one session's vector memory is absent from another's retrieval, and none exercises truncation of a non-zero-priority block. For a framework this widely deployed, the absent measurement is whether the default chat_history_token_ratio of 0.7 is a good split — it silently determines how much of every prompt is recent conversation versus durable memory, and no evidence for it appears in the repository.

11. For Your Own Build

Steal

• Self-truncating components. Ask each contributor to give back N tokens and let it choose how, instead of cutting the assembled string. Allowing a component to decline and be dropped is the right escape hatch — and ship at least one component that implements the shrink, or the contract is only a drop.
• One budget with an explicit split, rather than independent caps per subsystem that can jointly overflow.
• Overflow as the capture trigger. Coupling long-term writes to short-term pressure keeps the two tiers in one pipeline — provided you accept that quiet conversations write nothing.
• State the replacement contract in the prompt. If a condensation pass replaces rather than appends, say so in the instruction; the default model behaviour is to return a delta.
• Expose insert placement, since system-prompt versus user-message placement is a real cache-versus-freshness decision.
• A three-method block interface as an extension point — get, put, truncate is close to the minimum viable memory component contract.

Avoid

• Extraction with no provenance, when the source messages were available at extraction time.
• No tombstone under automatic re-extraction.
• Memory formation gated by conversation length.
• Wholesale condensation without a loss check.
• Two APIs sharing the word "memory" with very different guarantees.
• Writing a per-call filter into long-lived component state. Build the query's filters per call; a mutation that is correct once is wrong for every later caller.

Fit

Borrow:

• The atruncate contract and the budget split, with an implementation per block.
• The block interface as the shape of a pluggable memory component.
• The explicit full-replacement wording in any condensation prompt.

Do not copy:

• The absence of provenance, if you will ever need to explain a fact.
• Overflow-triggered capture as the only write path.
• Condensation without a structural-loss guard, if facts matter.

12. Open Questions

• Is chat_history_token_ratio = 0.7 right? It shapes every prompt and is undefended.
• Should facts carry the message ids they were extracted from? The information is present at extraction and discarded.
• What happens to a user-corrected fact when the next condensation runs?
• Should a short conversation be able to write long-term memory without an overflow?
• How do multiple blocks divide the long-term budget among themselves when several want more room?
• Is the priority order meant to truncate 1 first, as _truncate_memory_blocks does, or last, as the field description implies?

Appendix: File Index

• Core: llama-index-core/llama_index/core/memory/memory.py (Memory, BaseMemoryBlock, InsertMethod).
• Blocks: memory/memory_blocks/static.py, fact.py (FactExtractionMemoryBlock, extract and condense prompts), vector.py.
• Types: memory/types.py.
• Legacy window-management APIs: memory/chat_memory_buffer.py, chat_summary_memory_buffer.py, vector_memory.py, simple_composable_memory.py.

History

2026-09-15 — 0f43c00b… — 63 commits on, 2026-09-14; two touch memory: Memory accepts any AsyncDBChatStore (#22541) and SimpleChatStore persists non-ASCII unescaped (#22538). Read from a sparse checkout of the memory package, the chat stores and their tests; the screen covered the repository root and llama-index-core manifests — two build-time Makefiles, nothing auto-run, unpinned or inside the cooldown — and not the integration packages. Nothing was installed or run. The block code is unchanged since the previous pin, and three claims from the first reading did not survive it. atruncate is not implemented by any shipped block, so the self-truncation described as the design's best idea is a hook that drops a block whole, and default priority 0 exempts blocks from truncation. The report said there was no scope key; session_id is stamped on vector nodes and filtered on vector retrieval, through a filter mutated into the block instance that sticks to the first session. And removed facts returning from the same conversation was not possible as described: flushed messages are archived. No mark changes.

2026-07-27 — 199e9b5b… — first reading.