✦ Benchmarks, run honestly

How well does Eve remember?

We run Eve on the public memory benchmarks the field uses — each scored by the benchmark's own grader — and show it against the range of published competitor scores. Every number is backed by the unedited transcripts, so you can check each one yourself.

See where Eve stands How to verify it
Where Eve stands

Eve against the field, on each benchmark's own scoreboard

Reported memory-benchmark scores vary widely by grader and subset, so we show competitors as the range of their published numbers rather than one cherry-picked figure — and mark where Eve lands. Each card links to the raw transcripts behind Eve's score.

How we measured it

Same harness. Same grader. Same episodes.

The whole point of a benchmark is that someone else can reproduce it. So for each one we change nothing about how the test is scored — we only swap in Eve as the memory layer behind the agent.

The paper's own harness

We run the released task episodes through each benchmark's published harness — not a reimplementation. The questions, the order, and the scoring rubric are the paper's.

The agent does the thinking; Eve is the memory

A standard model answers each episode the way a normal user's agent would, with Eve connected as its memory layer — the same way you'd plug Eve into your own Claude or ChatGPT.

The paper's grader assigns the score

Final answers are scored by the benchmark's own grader at its pinned version. We don't grade ourselves — the rubric that ranks everyone else ranks us.

We change nothing on the benchmark's side — episodes, grader, and rubric run at the paper's pinned versions, and where a paper reports competitors under a specific internal model, we compare only against the rows on the same internal model Eve runs (GLM-5.1). Where the protocol leaves Eve's side open, we disclose exactly what we did rather than assume the flattering reading:

How Eve answers — both benchmarks. Every question gets a fresh conversation (batch = 1: no context or answer bleed across questions — the strict setting). Within that conversation, the answering agent runs one bounded agentic session (≤12 read-only steps; it may make several recalls and union them before answering) — it is not limited to a single retrieval call. We think one bounded session is the fair unit for an agentic memory system — it's how the product actually works — but on MEME that is a bigger retrieval budget than the paper's single-retrieval Table-4 answerer, so we say so here rather than let you find it in the traces. The ingest/memory side is identical to each paper's protocol. See a full answer session, recall-by-recall →

LOCOMO judging — the stricter ruler: a gpt-4o judge with no lean-to-yes and no answer-teaching. MEME answer model — claude-sonnet-4-6, as the paper's claude-sonnet-4 is retired from the API.

The honest number — and the honest method — is the one we publish.
Don't take our word for it

Here are the exact traces

Extraordinary numbers deserve scrutiny — so every episode is recorded in full. The harness calls Eve the same way any agent does, through our public connector, and logs exactly what went in and what came back: every ingested turn, every memory Eve formed, every question, Eve's answer, and the grade the paper's own grader gave it.

benchmark question → Eve (live API) → answer → paper's grader → score

Nothing summarized, nothing cherry-picked — read the full trace of every episode →. Independent verification with the benchmarks' authors is in progress.