Agent memory field guide: papers, security, benchmarks, and how Substrate is built

Shared by Pavel · Updated 2026-11-16

Start here

Agent memory ranges from lossy summaries and extracted facts to eidetic storage of exact messages, tool outputs and documents with timestamps and sources. Persistent memory can improve retrieval, but it is also an attack surface: untrusted content may later be read as trusted. Benchmarks have important limits, including reported LoCoMo answer-key errors and scores that are often not comparable across papers. “Eidetic” is a loose engineering term, not an established framing in the literature.

Topic map

Concepts

Agent memory can be episodic (what happened), semantic (facts), or procedural (how to do things). Other core concepts include provenance, temporal validity, contradiction handling, abstention, memory scope, shared versus private memory, and selective forgetting.

Papers

More

The guide links research including CoALA, LoCoMo, LongMemEval, MemoryAgentBench, MemGPT, Mem0, Zep, A-MEM, HippoRAG, TriMem and work on episodic memory. An, in a single-author ablation, reported verbatim chunks beating LLM-extracted artifacts by 15.9 points on LoCoMo and 22.0 points on LongMemEval-S; that is one paper’s result, not a general conclusion.

Talks

Talk coverage includes Chalef’s 2026 AI Engineer talk “Citation Needed,” about provenance and trust. Talk summaries are based on event pages rather than viewing the talks.

Security

More

Reported attacks include memory injection, tool poisoning, malicious issues, and data exfiltration. The dual-LLM pattern keeps untrusted content with a quarantined model that has no tools; the privileged model never sees that content. The Agents Rule of Two limits an agent to at most two of: processing untrusted inputs, accessing sensitive systems or private data, and changing state or communicating externally.

Benchmarks

More

LoCoMo tests long-conversation QA, event summarisation and multimodal dialogue; LongMemEval tests extraction, multi-session and temporal reasoning, knowledge updates, and abstention. MemoryAgentBench covers accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Mem0’s reported scores are its authors’ own: 66.88 ± 0.15 on LoCoMo and 21.1 on the MemoryAgentBench authors’ S* variant of LongMemEval. Scores from different papers are usually not comparable.

Open problems

The guide flags underscaled or saturating benchmarks, uncertain semantic value of metrics, dependence on judge models and backbones, and neglected maintenance costs.

History of machine memory

The guide treats agent memory as a developing field spanning persistent memory, retrieval, provenance, and security. Its start page distinguishes exact-source retention from lossy summaries and points to episodic memory as closer research vocabulary than “eidetic memory.”

Human memory

“Eidetic” is used loosely as an engineering analogy; the guide cautions that neuroscience terms are analogies drawn by agent-memory papers, not claims that LLM memory works like the brain.

Systems landscape

More

The guide describes systems with different approaches: MemGPT/Letta lets agents edit memory through tools and tiers; Mem0 extracts candidate facts and chooses ADD, UPDATE, DELETE or NOOP; Graphiti links derived entities and facts to raw episodes; TriMem links raw dialogue segments to facts and profiles.

Privacy and forgetting

More

Shared memory can leak information, and conversational memory brings deletion and retention concerns for raw and derived copies. Verbatim storage increases privacy exposure, while source tagging can help identify and remove poisoned entries.

Glossary

The glossary defines terms alphabetically and links entries to explanatory pages or sources. Its entries describe products and papers according to their authors, not as endorsements.

FAQ

The FAQ separates general agent-memory research from Substrate vendor material. It defines agent memory as what an agent keeps between steps or sessions so it does not start from zero, and points to CoALA’s working, episodic, semantic and procedural categories.

Substrate: what and why

More

Substrate is presented as a product built by the wiki’s author, with its vendor material kept separate from the neutral research corpus.

Substrate: architecture

More

Substrate’s described loop is capture with consent, distill source material into evidence-backed atoms, human correction, then agent recall. Its retrieval description—dense, lexical, graph and temporal retrieval followed by reranking, with local embeddings—is the author’s description; the public site does not detail retrieval.

Connections are approved through browser OAuth and can be revoked; agent keys are stored hashed. Model output is checked before storage, and the model does not choose record identifiers or account ownership. Sharing can be revoked.

The principles include making memory correctable, sharing it across agents with permissions, showing evidence rather than promising correctness, and combining benchmark numbers with day-to-day product use.