MRCR and long-context benchmark neighbors
Shared by Pavel · Updated 2026-11-12
MRCR tests whether a model can resolve references across repeated, multi-turn conversations; it is a synthetic long-context task, not a test of persistent memory.
MRCR: task, construction, and corrections
MRCR means Multi-Round Co-reference Resolution. Its conversations are generated synthetically and designed to test resolving co-references across multiple rounds.
OpenAI’s dataset card notes generation bugs; corrected versions were uploaded on 5 December 2025. Per the card, a generation bug put too many target needles in about 10% of datapoints and incorrect ground truth in about 5%. Scores published before that date may not be comparable.
How other long-context benchmarks differ
These benchmarks vary both in scale and in what “using long context” requires: from locating a planted fact to reasoning over scattered evidence or application-style tasks.
| Benchmark | Length or scale | Task design |
|---|---|---|
| Needle in a Haystack | Sweeps context length and needle depth. | Finds a short fact in unrelated text; variants include single-fact, multi-fact, and UUID-chain tasks. |
| RULER | Configurable; the summary gives no context-length range. | Synthetic benchmark with 13 tasks, rather than literal needle lookup alone. |
| NoLiMa | Evaluates models claiming at least 128K-token context. | Needle-in-a-haystack questions whose wording shares almost no words with the needle. |
| ∞Bench | Average data length above 100K tokens. | Synthetic and realistic tasks in English and Chinese; retrieving a few passages is insufficient. |
| HELMET | Up to 128K tokens. | Seven application-centric categories; evaluated 59 long-context models. |
| LongBench v2 | Contexts from 8K to 2 million words; 503 questions. | Multiple-choice questions across six categories. |
| BABILong | Splits extend up to 10 million tokens. | Twenty reasoning tasks with supporting facts scattered across long natural text. |
| Michelangelo | The summary gives no length range. | Minimal synthetic evaluations that require extracting latent structure from irrelevant context. |