I read through the skill file of graphify and ran some of the commands it was telling the agent to run. The vibe I got is there was a lot of structured information returned that the model (and harness) isn't set up to work with effectively.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
For the evals I've run (albeit, not super scientific), these memory systems did worse than the raw harness given ripgrep.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
Letting another LLM decide quality probably isn't the only meterstick you want, either.