5 min read

AI & Agents

Most AI Memory Benchmarks Are Marketing. Here's How I Report Holt's.

I nearly quoted a 97% score for Holt that was a tuning run on 101 questions. Here are the three numbers I publish, the three I refuse to, and the reason for each.

AI memory benchmarksLongMemEvalLoCoMo

Most AI Memory Benchmarks Are Marketing. Here's How I Report Holt's. · Products Decoded
By Debashis Nayak · Products Decoded3 Nov 2026

I almost put 97% on a page about Holt. I had the number in my head, it sounded great, and I went to check where it came from before using it. It came from a tuning run on a 101-question subset of LongMemEval, which I used to pick a default recall setting. A score you tune against is a score you cannot quote.

That same afternoon I found two more numbers I had been carrying around that were wrong. One of them I could not find a measurement for at all. So this post is the cleanup: what I publish for Holt, what I refuse to publish, and the test I now apply before any number leaves my laptop.

What are the Holt benchmark numbers I actually publish?

Three, all from Holt 0.19, all rerunnable by anyone in the public holt-os/holt-benchmarks repo.

BenchmarkHolt 0.19Note
LongMemEval91.2%Graded by GPT-4o. A second grader, Opus 5.5, gave 91.4%
LoCoMo88.2%
BEAM at 1M tokens79.6%

The 91.4% matters more than it looks. When two different models grade the same answers and land 0.2 points apart, the score is less likely to be an artefact of one grader’s habits. I still lead with the lower one, because if I have to pick, I pick the number that is harder to attack.

Rerunnable is the property I care about most. A benchmark claim that only I can reproduce is a story. One that anyone can rerun from a public repo is a measurement, and if I am wrong, somebody can show me.

Why is 97% not a Holt benchmark score?

Because of what it measured. The 97.0% was a tuning-run result on 101 questions from LongMemEval. I used that run to choose Holt’s default recall setting, which means the setting was selected because it did well on exactly those questions.

That is the oldest trap in benchmarking. Pick the configuration that wins on a small set, then report the winning score as if the set were a fresh exam. The honest number is the one from the full benchmark with the setting already locked in, and that number is the 91.2%.

There is a second problem, which is size. 101 questions is a small sample, and I would not trust a headline built on it even if I had not tuned against it. My guess is that the gap between 97.0 and 91.2 is a fair picture of how much selection flatters a score, but I have not isolated that, so treat it as a hunch.

What is the 97.8% figure?

Also not a score. 97.8% is how often my two graders agreed with each other.

I can see exactly how this one escapes. It is a real number, from a real run, sitting in the same output as the accuracy figures, and it is high. Grader agreement tells you the grading is consistent. It says nothing about how many questions Holt answered correctly. Put it in a deck next to the word “accuracy” and it will be read as accuracy, whether or not you meant it to be.

I report the grader agreement only when I am talking about how trustworthy the grading is, and I label it that way.

Where did 1.8 ms come from?

Nowhere I can point to. I believed Holt had a 1.8 ms latency, and when I went looking for the measurement, there was none. The only place that string turned up on my disk was inside the text of the BEAM dataset itself.

I am glad I looked. A latency number is the kind of claim that gets repeated, and one that nobody ever measured is just a rumour with units on it.

Here is the part that stings a little. Open-source Holt has no measured P95 recall latency at all, today. Measuring it is roughly a 20-minute job and it is still on my list. Until it is done, the correct statement is “not yet measured”, and that is what I say.

Is there any latency data at all?

Yes, but it belongs to a different product. Holt Teams is the closed, separate engine, and its early stress tests at 500K memories gave these results:

  • Recall P95 of 104 ms in an independent run and 118 ms in another, against a 150 ms budget.
  • Hook overhead of about 6 ms per turn.
  • Vector search P95 cut from 796 ms to 13.5 ms, using binary quantisation with an int8 rerank.

Those are stress-test numbers for the Holt Teams engine, not for the open-source Holt you can install. I attribute them to the Teams engine every time, and I keep them out of any sentence about the open-source release. Mixing them would be easy, since both products share a name, and it would let a number from one product vouch for the other.

My rule before any number goes public

After this cleanup I settled on four questions. If a number fails any of them, it stays in my notes.

  1. Is it from a full run, with settings locked beforehand? Tuning subsets do not count.
  2. Does it measure what the sentence says it measures? Accuracy, agreement and latency are different things that happen to sit near each other in a log.
  3. Can a stranger rerun it from a public repo? If not, say so, or leave it out.
  4. Does it belong to the product I am describing? Closed-engine data stays labelled as closed-engine data.

The rule also works from the reader’s side. When you see a memory benchmark score for any tool, ask which of those four it fails. Most fail the first one quietly, because tuning on a small set is easy and nobody advertises it.

A short checklist for your own results

  • Write down which questions you tuned on, then exclude them from the number you report.
  • Name the grader model next to the score, and run a second grader if you can afford it.
  • Label agreement rates as agreement rates.
  • If you never measured it, write “not measured”. It is a complete sentence.
  • Put the rerun instructions where the claim is.

I would rather have 91.2% that I can defend than 97% I have to explain. Whether that costs me attention in a field where bigger numbers travel faster, I honestly do not know yet. I would still take the trade.