Introducing memorybench by supermemory 🔬 The standard, open source evaluation framework for all context systems, across any benchmark. Easily run and compare multiple memory providers. batteries included - comes with a Web UI, CLI, checkpoints and everything else.

Over the last year, we have built several setups for running benchmarks. But we ran into the same problems. Setting up the benchmarks, but also checkpoints, comparisons, finding out the results, etc. is too time consuming.

This also makes it harder to do "benchmarkhacking", as any provider which is truly SOTA, would be good across all the dimensions: - Multiple benchmarks - Multiple judges / answering models - Cost - Latency reports. We give everyone the grounds to find out, easily.

It is fully open-source. You can view the results live in a web UI. Configure the answering / judge prompts Restart from checkpoints And view failed results, and exactly why they failed. Go star it :)

We took inspiration from OpenBench by @GroqInc, and have included full documentation for playing around with it. Go ahead and add providers / benchmarks as you see fit!

@supermemory This is great. Good work making it more open and easy to understand the options out there.

@supermemory good work

@supermemory We also are building a memory solution, and we have faced this issue ourselves, I can't thank you enough to make my life easier thanks brotha ❣️

@supermemory will be great to evaluate memora

@supermemory So where are the results?

@supermemory nice

@supermemory where the results are hosted

@supermemory This is real work. All the best.

@supermemory Lfg

@supermemory Alright we’ll be on it asap! Thanks @DhravyaShah

@supermemory This is exactly the kind of standardized tooling the AI/ML space needs. Having a unified way to evaluate context systems across different providers makes it way easier to make informed decisions instead of just going with whatever's trending.

@supermemory this guys knows how to make demo videos, sick!

@supermemory really cool

@supermemory King move! Nice one

@supermemory Good one

@supermemory Superb

@supermemory BASED drop over the holidays 🙌

@supermemory shared benchmarks are how infra categories mature. comparison creates progress not noise.

@supermemory Looks solid, especially the open-source aspect. The focus on simplifying benchmarks could save a lot of time for folks in the field. Nice work

@supermemory exciting!

@supermemory @lateinteraction @a1zhang its not exactly a long context benchmark but would still be interesting to see how RLM performs here