memnos is measured on the public long-term-memory benchmarks with a named judge and an open harness — and we ship the per-question predictions, so you can check every answer yourself. Reproduce any number in one command.
The 500-question run on Supermemory's open MemoryBench harness — gpt-4o as both answerer and judge. Retrieval isn't the bottleneck (MRR 0.79 / NDCG 0.80); memnos surfaces the right memory, and the answer model does the rest. Strongest on factual recall (assistant-facts 98%, user-facts 93%); weakest on preference (47%) — the full per-type breakdown and every prediction are in the repo.
The full conversation set, scored by a gpt-4o judge across
three independent from-scratch ingests (65 / 64 / 65) —
a band, not a single lucky run. Every prediction for every run is committed to the repo
under benchmarks/results/.
# reproduce it yourself — full conversation set, from scratch
$ python benchmarks/locomo_eval.py --sample-ids 0,1,2,3,4,5,6,7,8,9
gpt-4o judge : 64–65% # three independent ingests: 65 / 64 / 65
# LongMemEval, 500 questions, via Supermemory's MemoryBench harness
gpt-4o judge : 78.4% # predictions published
# every run's per-question predictions live in benchmarks/results/Plenty of memory products publish one big number with no judge, no rubric, and no way to re-run it. Here's the standard memnos holds itself to instead.
Every number says exactly who graded it and how — gpt-4o, with the rubric in the open. No model quietly grading its own homework.
Open any run under benchmarks/results/ and read what memnos actually answered for each question — and how the judge scored it.
The eval script ships in the repo. Swap the judge, change the rubric, re-run from scratch — and you get the same numbers we did.
We publish the same answers under strict, standard, and lenient grading — so no rubric can flatter the number without you seeing the full spread.
One set of answers · three rubrics · all published
The same memnos answers, on the same benchmark, score anywhere from ~44% to ~88% purely on how strictly you grade. A vendor showing a single big number is choosing the friendly end of that band and not telling you. We publish all of it and stand on the standard-judge result — so the number you reproduce is the number we lead with.
These numbers come from an engine that does no LLM call at query time — recall is vector + full-text + a local reranker, fully deterministic. So the score you reproduce from the repo is the behavior you get in production: same retrieval, same ranking, every run.
memnos answers yes to all three. Use them on every vendor you evaluate.
Who graded it — and is the rubric published? memnos: gpt-4o judge, rubric in the open, the full strict-to-lenient band shown.
Can I see the per-question predictions? memnos: yes — every run's predictions are committed to benchmarks/results/.
Can I re-run it myself? memnos: yes — the harness ships in the repo; clone it, run it from scratch, get the same band.
Strong numbers, an open harness, and per-question predictions you can read. Self-host it and check every claim on this site for yourself.