Agent memory field guide: papers, security, benchmarks, and how Substrate is built
Shared by Pavel · Updated 2026-11-16
Start here
Agent memory ranges from lossy summaries and extracted facts to eidetic storage of exact messages, tool outputs and documents with timestamps and sources. Persistent memory can improve retrieval, but it is also an attack surface: untrusted content may later be read as trusted. Benchmarks have important limits, including reported LoCoMo answer-key errors and scores that are often not comparable across papers. “Eidetic” is a loose engineering term, not an established framing in the literature.
Topic map
- Concepts: memory types, provenance, correction, forgetting and retrieval.
- Papers: research papers and short takes; the page count varies between versions.
- Talks: event-page summaries, not summaries from watching the talks.
- Security, benchmarks, open problems, history of machine memory, human memory, systems landscape, privacy and forgetting, glossary, FAQ: companion topics in the guide.
- Substrate: separate vendor material about what and why, architecture, security model, and principles and lessons.
Concepts
Agent memory can be episodic (what happened), semantic (facts), or procedural (how to do things). Other core concepts include provenance, temporal validity, contradiction handling, abstention, memory scope, shared versus private memory, and selective forgetting.
Papers
More
The guide links research including CoALA, LoCoMo, LongMemEval, MemoryAgentBench, MemGPT, Mem0, Zep, A-MEM, HippoRAG, TriMem and work on episodic memory. An, in a single-author ablation, reported verbatim chunks beating LLM-extracted artifacts by 15.9 points on LoCoMo and 22.0 points on LongMemEval-S; that is one paper’s result, not a general conclusion.
Talks
Talk coverage includes Chalef’s 2026 AI Engineer talk “Citation Needed,” about provenance and trust. Talk summaries are based on event pages rather than viewing the talks.
Security
More
Reported attacks include memory injection, tool poisoning, malicious issues, and data exfiltration. The dual-LLM pattern keeps untrusted content with a quarantined model that has no tools; the privileged model never sees that content. The Agents Rule of Two limits an agent to at most two of: processing untrusted inputs, accessing sensitive systems or private data, and changing state or communicating externally.
Benchmarks
More
LoCoMo tests long-conversation QA, event summarisation and multimodal dialogue; LongMemEval tests extraction, multi-session and temporal reasoning, knowledge updates, and abstention. MemoryAgentBench covers accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Mem0’s reported scores are its authors’ own: 66.88 ± 0.15 on LoCoMo and 21.1 on the MemoryAgentBench authors’ S* variant of LongMemEval. Scores from different papers are usually not comparable.
Open problems
The guide flags underscaled or saturating benchmarks, uncertain semantic value of metrics, dependence on judge models and backbones, and neglected maintenance costs.
History of machine memory
The guide treats agent memory as a developing field spanning persistent memory, retrieval, provenance, and security. Its start page distinguishes exact-source retention from lossy summaries and points to episodic memory as closer research vocabulary than “eidetic memory.”
Human memory
“Eidetic” is used loosely as an engineering analogy; the guide cautions that neuroscience terms are analogies drawn by agent-memory papers, not claims that LLM memory works like the brain.
Systems landscape
More
The guide describes systems with different approaches: MemGPT/Letta lets agents edit memory through tools and tiers; Mem0 extracts candidate facts and chooses ADD, UPDATE, DELETE or NOOP; Graphiti links derived entities and facts to raw episodes; TriMem links raw dialogue segments to facts and profiles.
Privacy and forgetting
More
Shared memory can leak information, and conversational memory brings deletion and retention concerns for raw and derived copies. Verbatim storage increases privacy exposure, while source tagging can help identify and remove poisoned entries.
Glossary
The glossary defines terms alphabetically and links entries to explanatory pages or sources. Its entries describe products and papers according to their authors, not as endorsements.
FAQ
The FAQ separates general agent-memory research from Substrate vendor material. It defines agent memory as what an agent keeps between steps or sessions so it does not start from zero, and points to CoALA’s working, episodic, semantic and procedural categories.
Substrate: what and why
More
Substrate is presented as a product built by the wiki’s author, with its vendor material kept separate from the neutral research corpus.
Substrate: architecture
More
Substrate’s described loop is capture with consent, distill source material into evidence-backed atoms, human correction, then agent recall. Its retrieval description—dense, lexical, graph and temporal retrieval followed by reranking, with local embeddings—is the author’s description; the public site does not detail retrieval.
Connections are approved through browser OAuth and can be revoked; agent keys are stored hashed. Model output is checked before storage, and the model does not choose record identifiers or account ownership. Sharing can be revoked.
The principles include making memory correctable, sharing it across agents with permissions, showing evidence rather than promising correctness, and combining benchmark numbers with day-to-day product use.