Agent Memory Testbench

Stuff every ingested turn into the prompt up to the model context limit. Ceiling baseline for what an LLM could answer with perfect recall.

Last N turns across all sessions. Cheapest baseline. Approximates a chatbot with no memory beyond its current scrollback.

Embed every turn with text-embedding-3-large, retrieve top-k by cosine similarity. The 'put it in a vector DB' default.

Pure-Python lexical baseline using rank-bm25. Old-school keyword search, what Google did before vectors.

Reciprocal Rank Fusion (k=60) over a vector ranking and a BM25 ranking. Captures both meaning matches and exact-keyword matches.

Hypothetical Document Embeddings: ask the LLM to write a plausible answer first, embed THAT, retrieve real turns by similarity, then answer.

Build a one-page JSON profile of the user via Haiku once before recall, then stuff that profile into every answer's system context.

Generative-agents pattern (Park et al. 2023): every N sessions write a synthetic LLM-authored summary, index it alongside raw turns.

Hierarchical k-means clustering with LLM cluster summaries (Sarthi et al. 2024). 4 levels deep, branch=4. Like a table of contents for the chat.

Karpathy's LLM-maintained wiki: every session triggers an LLM call that creates or appends entity pages with [[wikilinks]] and [session=...] citations. Periodic lint pass merges duplicates and resolves contradictions.

HippoRAG 2 (Gutierrez et al., ICML 2025): hippocampus-inspired memory. Haiku open-IE extracts (subject, predicate, object) triples per session into a networkx graph. Synonym edges link near-duplicate entities. At recall, query embeddings seed personalized PageRank over the graph; passages are scored by PPR mass on their entities.

A-MEM (NeurIPS 2025, Xu et al.): the LLM emits structured memory notes (content + keywords + context + tags) per session, embeds them, and a periodic link-evolution pass adds directed edges between related notes. Retrieval fans out from each top-k hit to its linked neighbors.

Quantum-Inspired Semantic Similarity: reranks naive_vector's candidates by quantum-state fidelity Tr(rho_q rho_d) = cosine squared over the same embeddings. Pure NumPy, no new deps. An optional multi-query superposition mode adds interference cross-terms that classical rank fusion cannot express.

Simulated Quantum Reranker: runs a real SWAP-test circuit on the Qiskit Aer simulator to estimate each query-document overlap. Embeddings are PCA-reduced to 2^n_qubits dims, amplitude-encoded, and compared in exact statevector mode. The accuracy cost of that dimensionality reduction is reported honestly.

Mem0 SDK (v2), leveled to the harness: Chroma vector store, OpenAI embeddings, Anthropic Sonnet for fact extraction during ingest (same model as the pure-Python baselines).

Zep's Graphiti: temporal knowledge graph with valid_at/invalid_at edges. Each session ingests as one episode. Searches return time-aware facts.

Zep's Graphiti running on FalkorDB (a Redis-based, GraphBLAS graph engine) instead of Neo4j. Identical ingest and recall to graphiti; the graph database is the only variable, so the latency and cost deltas isolate the engine and put FalkorDB's headline latency claims to an independent test.

Open-source knowledge-graph memory. Cognify extracts entities/relationships into a graph; search answers via the graph as context.

LangChain's memory store: extracts facts as conversations happen, stores them in LangGraph's InMemoryStore, recalls by similarity.

SQL-native agent memory. Stores extracted facts in Postgres for SQL-style recall. Memori 3.x with autocommit BYODB conn.