KnowMe-Bench comparison
Shared by Pavel · Updated 2026-11-12
The reported run puts Substrate and a strengthened Mem0 OSS re-run roughly even on KnowMe-Bench: 66.59% versus 65.09%, a 1.5-point difference. It does not establish broad superiority for Substrate.
What it establishes
- Both systems were evaluated with the same answer model and judge, making this a like-for-like comparison on those parts of the setup.
- KnowMe-Bench tests person understanding from autobiographical narratives, with evidence-linked questions; the result is evidence about performance on that benchmark, not agent memory overall.
- The Mem0 OSS baseline was strengthened for the re-run. The answer model was also stronger than the models used in Mem0’s own paper runs, so the reported scores are not directly comparable with those published results.
What limits interpretation
- This was one internal run, reported by Substrate, and has not been independently reproduced. The 1.5-point gap therefore does not establish a reliable advantage.
- Benchmark scores do not cover other important dimensions, including exact evidence and provenance, deletion, adversarial writes, cost, latency, and held-out evaluation.
- Substrate’s stated approach is to treat benchmark numbers alongside daily product experience, since numbers can be noisy or flawed and experience can also mislead.