MRCR and long-context benchmark neighbors

Shared by Pavel · Updated 2026-11-12

MRCR tests whether a model can resolve references across repeated, multi-turn conversations; it is a synthetic long-context task, not a test of persistent memory.

MRCR: task, construction, and corrections

MRCR means Multi-Round Co-reference Resolution. Its conversations are generated synthetically and designed to test resolving co-references across multiple rounds.

OpenAI’s dataset card notes generation bugs; corrected versions were uploaded on 5 December 2025. Per the card, a generation bug put too many target needles in about 10% of datapoints and incorrect ground truth in about 5%. Scores published before that date may not be comparable.

How other long-context benchmarks differ

These benchmarks vary both in scale and in what “using long context” requires: from locating a planted fact to reasoning over scattered evidence or application-style tasks.

BenchmarkLength or scaleTask design
Needle in a HaystackSweeps context length and needle depth.Finds a short fact in unrelated text; variants include single-fact, multi-fact, and UUID-chain tasks.
RULERConfigurable; the summary gives no context-length range.Synthetic benchmark with 13 tasks, rather than literal needle lookup alone.
NoLiMaEvaluates models claiming at least 128K-token context.Needle-in-a-haystack questions whose wording shares almost no words with the needle.
∞BenchAverage data length above 100K tokens.Synthetic and realistic tasks in English and Chinese; retrieving a few passages is insufficient.
HELMETUp to 128K tokens.Seven application-centric categories; evaluated 59 long-context models.
LongBench v2Contexts from 8K to 2 million words; 503 questions.Multiple-choice questions across six categories.
BABILongSplits extend up to 10 million tokens.Twenty reasoning tasks with supporting facts scattered across long natural text.
MichelangeloThe summary gives no length range.Minimal synthetic evaluations that require extracting latent structure from irrelevant context.