This number comes from a paper we did not set. We ran VENETA WorldModel's memory and self-improvement layer through LongMemEval-S as it is: all 500 questions, Claude Opus 5.5 as the writer, grading by the benchmark's own GPT-4o judge, three runs on each arm.
- Memory layer on 94.5 %, mean of three runs [93.6–95.2]
- Memory layer off 19.7 %, mean of three runs [19.2–20.2]
The layer was built for on-premises operations: data that cannot leave the building, a local model inside the customer's network writing the answers, and decisions rather than conversations as the thing worth remembering. No memory benchmark measures that environment, so we built our own. The question here was the opposite one. Swap the writer for a frontier model, hand the layer a paper written for ordinary consumer use, and see what happens.
Where it is strong and where it is not
Knowledge update 98.3 (78 questions), what the assistant said 97.6 (56), what the user said 96.7 (70), temporal reasoning 93.2 (133), multi-session reasoning 89.7 (133). Single-session preference is a perfect score across all three runs, but on 30 questions — too small a denominator to read as evidence of perfection. The column we are watching is multi-session reasoning, where the denominator is 133.
Both arms get the same budget
The arm with the layer on reads a recall block of 60,000 characters. The arm with it off gets the tail of the same conversation, cut to 60,000 characters. Conversations in this benchmark average 489,731 characters — three books' worth — and not one of them is shorter than 60,000, so the off arm fills its budget on every question.
19.7 % should not be read on its own
Close to a third of what the off arm gets right is the abstention set: 30 questions that ask whether a model says it does not know when the conversation never said. An arm that has seen almost nothing answers "I don't know" to nearly everything, so it scores full marks on that column alone. Across the other 470 questions the off arm is at 14.6 % [14.0–15.1].
What we do not claim from this
- We do not say we are better than other memory products. We have not matched conditions against them, so we are not in a position to compare.
- We do not say this benchmark measures memory in full. LongMemEval looks at long conversations between a person and an assistant. Whether a record of what an operator approved, rejected and corrected survives is what our own benchmark asks.
- 94.5 % is not a number from the product's default setting. The product's default recall budget is 3,000 characters, sized for an operator's decision record.
The conditions, the per-run counts behind every ability, and a walk through each question the layer got wrong are on project veneta: the LongMemEval results.