Benchmarks
Shared by Pavel · Updated 2026-11-12
Memory benchmarks test conversation recall and reasoning over evolving histories; security benchmarks test attacks such as poisoning, instruction injection, and information leakage. Their constructions and reported results differ substantially.
Memory benchmarks
More: MRCR and long-context benchmark neighbors
- LoCoMo uses long conversations—averaging 300 turns and 9,000 tokens, with up to 35 sessions—for question answering, event summarisation, and multimodal dialogue. The conversations are synthetic and then human-verified. It does not test knowledge updates; its conversations can fit within a context window, so a full-context baseline can beat memory systems.
- LongMemEval poses 500 questions over scalable chat histories, testing information extraction, multi-session and temporal reasoning, knowledge updates, and abstention. Its abstention set includes 30 false-premise questions. The small setting is about 115,000 tokens and the medium setting about 1.5 million.
- MemoryAgentBench reformulates existing datasets and adds EventQA and FactConsolidation in an incremental, multi-turn setup. It tests accurate retrieval, test-time learning, long-range understanding, and selective forgetting. FactConsolidation uses counterfactual edit pairs in which later facts contradict earlier ones; the benchmark reports that all methods fail on its multi-hop forgetting version. Smaller chunks can help retrieval but hurt long-range understanding.
- MMRC broadens memory evaluation to real-world multimodal conversations: 5,120 conversations and 28,720 manually labelled questions across six abilities, including information updates, image management, and memory recall.
Security benchmarks
- AgentDojo sets up 97 realistic tasks with 629 security test cases in an extensible environment. InjecAgent has 1,054 cases, 17 user tools, 62 attacker tools, and 30 agents; the reported 24% vulnerability rate is specifically for ReAct-prompted GPT-4.
- Agent Security Bench (ASB) combines 10 scenarios, over 400 tools, 27 attack and defence types, and 13 model backbones. Its reported highest average attack success is 84.30%; results are tied to that benchmark setup, not a universal agent failure rate. (more: KnowMe-Bench comparison)
- MemPoison contains 1,227 hand-validated cases across four attack types, three injection channels, and three memory substrates. Its tiers distinguish single-record corruption, multi-record compositional corruption, and context-triggered attacks. AgentLeak tests 1,000 scenarios of shared-memory privacy risk; it reports 68.8% leakage in inter-agent messages, while final-output leakage is lower in multi-agent than single-agent mode (27.2% vs. 43.2%).
Limits on reported results
- Mem0’s authors report 66.88 ± 0.15 on LoCoMo, but the comparison is author-run and judge-dependent. A full-context baseline scored 72.90 ± 0.19 in the same table. Penfield Labs reported answer-key errors that could affect scores; its audit is a company blog post and was not independently replicated.
- Mem0’s 21.1 is on the MemoryAgentBench authors’ S\* variant of LongMemEval, not a directly comparable LongMemEval score.
- LLM-as-judge scores depend on the judge model and prompt. Benchmark results can also be distorted by answer-key errors, lexical overlap, and history length; scores from different papers are generally not comparable.
- Security attack-success figures are controlled-study results, not estimates of real-world prevalence. For example, the listed AgentLeak percentages describe its 1,000 tested scenarios, and the InjecAgent figure is specific to one prompting setup.